サーバ上で取ったJSONログの解析にjqを使うと便利
通常のWEBサーバのログ以外に独自でJSON形式のログを取ったりしているのだが、
その解析に「jq」というツールが便利
インストール
[shell]
# cd /usr/local/bin
# wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -O jq
[/shell]
こういう感じのJSONログがあったとする
[shell]
{"time":1493398942,"url":"hoge.kunikiya.jp/1","ip":"66.249.79.78","ua":"Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)","lang":""}
{"time":1493398944,"url":"hoge.kunikiya.jp/1","ip":"207.46.13.51","ua":"Mozilla\/5.0 (compatible; bingbot\/2.0; +http:\/\/www.bing.com\/bingbot.htm)","lang":""}
{"time":1493398949,"url":"hoge.kunikiya.jp/1","ip":"40.77.167.53","ua":"Mozilla\/5.0 (compatible; bingbot\/2.0; +http:\/\/www.bing.com\/bingbot.htm)","lang":""}
{"time":1493398951,"url":"hoge.kunikiya.jp/1","ip":"66.249.79.99","ua":"Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/41.0.2272.96 Mobile Safari\/537.36 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)","lang":""}
[/shell]
とりあえずJSONをパースして見る
[shell]
$ cat test.log | jq .
{
"time": 1493398942,
"url": "hoge.kunikiya.jp/1",
"ip": "66.249.79.78",
"ua": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"lang": ""
}
{
"time": 1493398944,
"url": "hoge.kunikiya.jp/1",
"ip": "207.46.13.51",
"ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",
"lang": ""
}
{
"time": 1493398949,
"url": "hoge.kunikiya.jp/1",
"ip": "40.77.167.53",
"ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",
"lang": "Ja-Jp"
}
{
"time": 1493398951,
"url": "hoge.kunikiya.jp/1",
"ip": "66.249.79.99",
"ua": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"lang": ""
}
[/shell]
UAが「hogehoge」のものだけ
[shell]
$ cat test.log | jq 'select(.ua == "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)")'
{
"time": 1493398944,
"url": "hoge.kunikiya.jp/1",
"ip": "207.46.13.51",
"ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",
"lang": ""
}
{
"time": 1493398949,
"url": "hoge.kunikiya.jp/1",
"ip": "40.77.167.53",
"ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",
"lang": "Ja-Jp"
}
[/shell]
langに値があるものだけ
[shell]
$ cat test.log | jq 'select(.lang != "")'
{
"time": 1493398949,
"url": "hoge.kunikiya.jp/1",
"ip": "40.77.167.53",
"ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",
"lang": "Ja-Jp"
}
[/shell]
LIKE検索はこのようにできるらしいが残念ながら手元の環境が古いサーバだったのでバージョン1.5以上じゃないと使えなかった
[shell]
cat sampling.log* | jq '.[] | select(.ua | startswith("Mozilla"))'
[/shell]
jq1.4以下だとLIKE検索するのに良さそうなものがなかったのでgrepしてから処理
[shell]
cat sampling.log* | grep "Mozilla" | jq 'select(.lang == "")'
[/shell]
awkと組み合わせて集計に使ったり
[shell]
cat test.log | jq 'select(.lang == "")' | jq .ip | awk '{ips[$0]++}END{for(ip in ips){print ip, ips[ip]}}' | sort -k2 -n
"207.46.13.51" 1
"66.249.79.78" 1
"66.249.79.99" 1
[/shell]
jqで文字列のlike検索をする
http://qiita.com/drwtsn64/items/fddc69e984f90b53d105
他にも色々フィルタや演算子が揃ってて結構使える感じだった
詳しくはここがまとまってる