PeterVandivier
I've received a json payload that I know to be a modification of a certain document.json; but it looks like this (after it's been prettified):
```json
{
"z":"oof",
"b": [
1,
2,
{
"d": "bar",
"e": [
0,
1
]
}
],
"f": {
"g": "bork"
},
"i": "zap",
"a": "foo"
}
```
I've saved this payload as document2.json. Now I would like to figure out exactly what changed with my document since I last looked at it in [Question 888][1].
@@@ question 888
To see the change between the original document and the current payload, I plan to...
1. extract all leaf paths from both documents
2. query the documents for the leaf path values
3. save the output as sorted key-value-pairs
4. diff the sorted KVP outputs
I would like to read lines from a file and execute them as queries using `jq`.
```bash
#!/usr/bin/env bash
jq_leaf_paths='paths(.==null or scalars) | map(if type == "number" then "[\(tostring)]" else "."+. end) | join("")'
keys=$(jq "$jq_leaf_paths" document2.json | sort)
echo '' > doc2.kvp
for key in $keys; do
value=$(jq $key document2.json)
echo "$key = $value" >> doc2.kvp
done
```
From the above script, I'm hoping to get the following output for step 3...
```
".a" = "foo"
".b[0]" = 1
".b[1]" = 2
".b[2].d" = "bar"
".b[2].e[0]" = 0
".b[2].e[1]" = 1
".f.g" = "bork"
".i" = "zap"
".z" = "oof"
```
...but instead I'm getting...
```
".a" = ".a"
".b[0]" = ".b[0]"
".b[1]" = ".b[1]"
".b[2].d" = ".b[2].d"
".b[2].e[0]" = ".b[2].e[0]"
".b[2].e[1]" = ".b[2].e[1]"
".f.g" = ".f.g"
".i" = ".i"
".z" = ".z"
```
So it looks like the leaf path query is getting read from the file properly, but isn't actually getting executed against the original document again inside the loop. How can I fix this? Plz halp!
[1]: https://topanswers.xyz/nix?q=888
Top Answer
PeterVandivier
While I'm not 💯% on the internals of _why_ this works, deferring execution with [`eval`][1] seems to do the trick.
Additionally, the `jq_leaf_paths` query requires a modification to handle for special characters in the event that they are present.[^n1]
```bash
#!/usr/bin/env bash
jq_leaf_paths='paths(.==null or scalars) | map(if type == "number" then "[\(tostring)]" elif test("[^a-zA-Z0-9_]") then ".\"\(.)\"" else "."+. end) | join("")'
keys=$(jq "$jq_leaf_paths" document2.json | sort)
echo '' > doc2.kvp
for key in $keys; do
value=$(eval jq $key document2.json)
echo "$key = $value" >> doc2.kvp
done
```
This produces the desired output. Running the same transform against the original document.json allows us to then diff the 2 sorted key-value-pair outputs and reveals the following differences using `git diff --unified=0 doc1.kvp doc2.kvp`...
```diff
diff --git a/doc1.kvp b/doc2.kvp
index a59495b..752c902 100644
--- a/doc1.kvp
+++ b/doc2.kvp
@@ -7,2 +7,2 @@
-".f.g" = "baz"
-".h" = null
+".b[2].e[1]" = 1
+".f.g" = "bork"
@@ -9,0 +10 @@
+".z" = "oof"
```
...which matches nicely with a deep manual read of each object
1. the array at `.b` (index 2) `.e` had an element added with value 1
2. `.f.g` changed values from "baz" to "bork"
3. element `.z` was added
4. element `.h` has a value `null` declared in document.json but is undeclared in the modified payload
[^n1]: The addition of `elif test("[^a-zA-Z0-9_]") then ".\"\(.)\""` will quote-escape-wrap index keys with special characters. A practical example would be "`@timestamp`" or "`@version`" fields present in ElasticSearch logs.
[1]: https://www.unix.com/man-page/posix/1posix/eval/
Answer #2
Jack Douglas
You can diff two files in a single command by getting the sorted full paths (including values) for each leaf node:
```shell
diff \
<(echo '{ "a": "foo", "b": [ 1, 2, { "d": "bar" } ] }' |
jq -S |
jq -c 'paths(.==null or scalars) as $path | getpath($path) as $v | $path | [.,$v]') \
<(echo '{ "b": [ 1, 2, { "d": "baz" } ], "a": "foo" }' |
jq -S |
jq -c 'paths(.==null or scalars) as $path | getpath($path) as $v | $path | [.,$v]')
4c4
< [["b",2,"d"],"bar"]
---
> [["b",2,"d"],"baz"]
```
The path strings can then be used to traverse the original JSON in `jq` again if required:
```shell
echo '{ "a": "foo", "b": [ 1, 2, { "d": "bar" } ] }' |
jq 'getpath(["b",2,"d"])'
"bar"
```