jq

由 datahunter 在三, 20/12/2017 - 16:08 發表

最後更新: 2023-07-04

介紹

jq is sed for JSON (Command-line JSON processor)

jq is written in portable C, and it has zero runtime dependencies.

HomePage: https://stedolan.github.io/jq/

Default: running the filter for each JSON object in the input

Install

apt-get install jq

dnf install jq

Opts

--raw-output / -r

if the filter´s result is a string then it will be written directly to stdout
rather than being formatted as a JSON string with quotes.

--compact-output / -c

JSON object on a single line

--null-input / -n

Don ´t read any input at all! Instead, the filter is run once using null as the input.

i.e.

jq -nr 'now'

1696493958.188751

Json file

boxs.json

{"Boxs":[
{"x": 1, "y": 2, "z": 3},
{"x": 4, "y": 5, "z": 6},
{"x": 7, "y": 8, "z": 9}]}

Usage

Pretty-print

# The simplest filter is "." which echoes its input, but pretty-printed:

echo '{"hello":{ "greetings":"to you"}}' | jq .

jq . file.json

* . refers to the input

Notes: keep colors when piping "jq" output to "less"

By default, jq outputs colored JSON if writing to a terminal.

You can force it to produce color even if writing to a pipe or a file using -C

curl -s http://localhost:9200/_nodes | jq -C .nodes | less -r

Field

single field: '.expression'

# ".field" which pulls "field" out of each record:

i.e.

echo '{"hello":{ "greetings":"to you"}}' | jq .hello

{
  "greetings": "to you"
}

nested data: '.expression.expression'

i.e.

echo '{"hello":{ "greetings":"to you"}}' | jq .hello.greetings

"to you"

取出兩個 Field: '.foo, .bar'

i.e.

echo '{"foo": 42, "bar": "string", "baz": true}' | jq '.foo, .bar'

42
"string"

P.S. 在 list of array 時寫法有所不同

jq '.Boxs[] | .x, .y' boxs.json

jq '.Boxs[].x, .Boxs[].y' boxs.json

Array

Array Index

# 0, 1, 2 ... *

.[N]

'length'

echo '["a","b","c","d","e"]' | jq 'length'

array - '[]'

jq '.Boxs' boxs.json # a array of object

jq '.Boxs[]' boxs.json # 一個個 object

jq '.Boxs[]' boxs.json

--slurp/-s:

read the entire input stream into a large array and run the filter just once.
(Default: running the filter for each JSON object in the input)

jq '.Boxs[]' boxs.json | jq -s

jq '.Boxs[]' boxs.json | jq [.]

Go “into” the array

.[]

.[] = Array/Object Value Iterator

有 go “into” Array/Object 的作用

.[]?

Like .[], but no errors will be output if . is not an array or object.

Slice - "[start-index:end-index]"

echo '["a","b","c","d","e"]' | jq '.[2:4]'

.[-3:-1] # length = 2. 獲得 [-3], [-2]
.[-3:] # length = 3. 獲得 [-3], [-2], [-1]

Labels

Adding labels to identifier values. To make this output easier to read

{<label>: <expression>, <label>: <expression>}

e.g.

Sort

sort
sort_by(.foo)

sort

sort a array.

Order

null
false / true
numbers
strings
arrays
objects

echo '[8,3,null,6]' | jq sort -c

[null,3,6,8]

'reverse'

echo '[8,3,null,6]' | jq sort | jq -c 'reverse'

[8,6,3,null]

sort_by(.foo)

compares two elements by comparing the result of foo on each element.

[{"foo":4, "bar":10}, {"foo":3, "bar":100}, {"foo":2, "bar":1}]
=> [{"foo":2, "bar":1}, {"foo":3, "bar":100}, {"foo":4, "bar":10}]

map

map(filter) / map_values(x)

run filter for each element of the input array, and return the outputs in a new array.

filter

has,
in,
select

select(boolean_expression)

echo '[1,2,3]' | jq -c 'map(select(. >= 2))'

[2,3]

echo '["a", "b", "c"]' | jq -c 'map(select(. >= "b"))'

["b","c"]   # ASCII a=141, b=142

# 由於用了 ".[]" 所以不用 map()

echo '[{"id": "first", "val": 1}, {"id": "second", "val": 2}]' | jq -c '.[] | select(.id == "second")'

{"id":"second","val":2}

has(key)

# returns whether the input object has the given key

echo '[{"foo": 42}, {"bar":37}]' | jq -c 'map(has("foo"))'

[true,false]

# returns array has an element at the given index.

echo '[["a", "b", "c"], ["e", "f"]]' | jq -c 'map(has(2))'

[true,false]

echo '[["a", "b", "c"], ["e", "f"]]' | jq .[0][2]

"c"

String

split(D) -> Array

cat list.txt | jq '.RecoveryPoints[] | .CreationDate|split("T")[0]' | less

cat list.txt | jq '.RecoveryPoints[] | .CreationDate[0:10]' | less

Date

jq -nr 'now'

1696493958.188751

jq -nr 'now | todate'

2023-10-05T08:48:06Z

jq -nr '"2023-10-05T08:48:06Z" | fromdate'

1696495686

strftime(fmt) # builtin formats a time (GMT) with the given format. P.S. strflocaltime

jq -nr 'now | strftime("%Y-%m-%dT%H:%M:%S")'

2023-10-05T08:19:53

日期轉 unix timestamp

* 由於 fromdate 只認 "%Y-%m-%dT%H:%M:%SZ" 格式(ISO 8601), 所以唔可以用它轉 timestamp

jq -nr '"2022-09-11T01:00:00+08:00"[0:19]+"Z"'

2022-09-11T01:00:00Z

Example

#1 找出 array 內每個 item 的 feild

ip-ranges.json

[
  {
    "ip_prefix": "3.2.34.0/26",
    "region": "af-south-1",
    "service": "AMAZON",
    "network_border_group": "af-south-1"
  },
  {
    "ip_prefix": "3.5.140.0/22",
    "region": "ap-northeast-2",
    "service": "AMAZON",
    "network_border_group": "ap-northeast-2"
  },
  ...
]

# "| .text" filter by text

cat ip-ranges.json | jq .prefixes | jq ".[] | .ip_prefix"

Notes

jq ".[] | .ip_prefix" 可以簡寫成 jq ".[].ip_prefix"

P.S.

AWC CloudFront 的 IP List: https://ip-ranges.amazonaws.com/ip-ranges.json

#2 在 elasticsearch 找出每個 node

curl -s http://localhost:9200/_nodes/ | jq '.nodes | keys'

# node 的基本資料

curl -s http://localhost:9200/_nodes/ | jq '.nodes | .[] | .name, .ip, .version, .roles, .attributes'

#3 Remove Old AWS Backup

# 獲得 resource-arn

aws backup list-protected-resources

# 找出 recovery-points

aws backup list-recovery-points-by-resource \
--resource-arn "arn:aws:ec2:ap-east-1:..." > backup_list.txt

# Filter 出 90 日前的 Backup

cat list.txt | jq '.RecoveryPoints[] | select(.CreationDate[0:19]+"Z" | fromdate < (now - 7776000))' > old_backup.txt

cat old_backup.txt | jq '.RecoveryPointArn' > delete_list.txt

delete_bak.sh

#!/bin/bash

VAULT="VAULT_NAME"

while read line
do
        aws backup delete-recovery-point \
                --backup-vault-name "$VAULT" \
                --recovery-point-arn "${line:1:-1}"
done < "delete_list.txt"

瀏覽次數： 75735

夢想家

jq

介紹

目錄

Install

Opts

Json file

Usage

Sort

map

String

Date

Example