サーバ上で取ったJSONログの解析にjqを使うと便利
通常のWEBサーバのログ以外に独自でJSON形式のログを取ったりしているのだが、
その解析に「jq」というツールが便利
インストール
|
1 2 |
# cd /usr/local/bin # wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -O jq |
こういう感じのJSONログがあったとする
|
1 2 3 4 |
{"time":1493398942,"url":"hoge.kunikiya.jp/1","ip":"66.249.79.78","ua":"Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)","lang":""} {"time":1493398944,"url":"hoge.kunikiya.jp/1","ip":"207.46.13.51","ua":"Mozilla\/5.0 (compatible; bingbot\/2.0; +http:\/\/www.bing.com\/bingbot.htm)","lang":""} {"time":1493398949,"url":"hoge.kunikiya.jp/1","ip":"40.77.167.53","ua":"Mozilla\/5.0 (compatible; bingbot\/2.0; +http:\/\/www.bing.com\/bingbot.htm)","lang":""} {"time":1493398951,"url":"hoge.kunikiya.jp/1","ip":"66.249.79.99","ua":"Mozilla\/5.0 (Linux; Android 6.0.1; Nexus 5X Build\/MMB29P) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/41.0.2272.96 Mobile Safari\/537.36 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)","lang":""} |
とりあえずJSONをパースして見る
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
$ cat test.log | jq . { "time": 1493398942, "url": "hoge.kunikiya.jp/1", "ip": "66.249.79.78", "ua": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "lang": "" } { "time": 1493398944, "url": "hoge.kunikiya.jp/1", "ip": "207.46.13.51", "ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", "lang": "" } { "time": 1493398949, "url": "hoge.kunikiya.jp/1", "ip": "40.77.167.53", "ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", "lang": "Ja-Jp" } { "time": 1493398951, "url": "hoge.kunikiya.jp/1", "ip": "66.249.79.99", "ua": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "lang": "" } |
UAが「hogehoge」のものだけ
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
$ cat test.log | jq 'select(.ua == "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)")' { "time": 1493398944, "url": "hoge.kunikiya.jp/1", "ip": "207.46.13.51", "ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", "lang": "" } { "time": 1493398949, "url": "hoge.kunikiya.jp/1", "ip": "40.77.167.53", "ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", "lang": "Ja-Jp" } |
langに値があるものだけ
|
1 2 3 4 5 6 7 8 9 |
$ cat test.log | jq 'select(.lang != "")' { "time": 1493398949, "url": "hoge.kunikiya.jp/1", "ip": "40.77.167.53", "ua": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", "lang": "Ja-Jp" } |
LIKE検索はこのようにできるらしいが残念ながら手元の環境が古いサーバだったのでバージョン1.5以上じゃないと使えなかった
|
1 |
cat sampling.log* | jq '.[] | select(.ua | startswith("Mozilla"))' |
jq1.4以下だとLIKE検索するのに良さそうなものがなかったのでgrepしてから処理
|
1 |
cat sampling.log* | grep "Mozilla" | jq 'select(.lang == "")' |
awkと組み合わせて集計に使ったり
|
1 2 3 4 |
cat test.log | jq 'select(.lang == "")' | jq .ip | awk '{ips[$0]++}END{for(ip in ips){print ip, ips[ip]}}' | sort -k2 -n "207.46.13.51" 1 "66.249.79.78" 1 "66.249.79.99" 1 |
jqで文字列のlike検索をする
http://qiita.com/drwtsn64/items/fddc69e984f90b53d105
他にも色々フィルタや演算子が揃ってて結構使える感じだった
詳しくはここがまとまってる