Add --output-file flag for writing values to file

[thaliaarchi]

This is a further development on @christf's #3367 (feat: allow writing to output file), which handles some edge cases not considered.

Other uses of stdout, such as tests, fuzzers, disassembly, and debug traces, remain directed to stdout; only JSON values are written to the output file. Consequently, users of --debug-dump-disasm and --debug-trace can now separate the debug and JSON outputs. On Windows, the output file is marked as binary when --binary is passed.

This incorporates the commit from #3367, but with minor fixes squashed in.

Fixes #2418
Closes #3367

[thaliaarchi]

Well that was a lot of back and forth with CI to identify the problem with Windows 😅. I'm confident in this now.

[christf]

How can this be progressed?

[thaliaarchi]

@wader Any more thoughts on this?

[wader]

is there any concerns to think about if output file is one of the input files? will get truncated before reading? streaming mode?

[thaliaarchi]

I don't think we can give a good experience for this, as the input files are opened after the output file is opened. Perhaps there's some way the file descriptors can be opened with copy-on-write semantics, so the input can still read the existing contents?

I should write a test for this.

[thaliaarchi]

I think that opening the file for reading, then opening it again for writing and truncating it, will allow the read fd to read the original contents. But we open them in the other order and inputs are opened lazily, so I don't know how to do this without eagerly opening all inputs, since they could all alias with sym/hardlinks. I'm skeptical it's worth it. Thoughts?

[wader]

Yeap seems messy :( tiny bit worried someone will try to use this as a "--in-place" workaround and be disappointed :)

[thaliaarchi]

Let's just add --in-place. I wouldn't mind implementing that. No need to complicate this so much for a mistake.

[Pandapip1]

What about storing the output file path and lazily opening the file when the full output is ready to be written?

[christf]

What about storing the output file path and lazily opening the file when the full output is ready to be written?

This approach would mean all output must be kept in a buffer before it can be written. As such, streaming becomes impossible and the amount of data to be written is constrained by the amount of memory of the machine.I do not think there will be a great approach here. I can see these options

• do not stream as @Pandapip1 has described (this has the above consequence)
• write to a temporary file instead and rename the file after opening the last input file (yikes)
• open all input files right at the start before opening the output as @thaliaarchi has described

I am not sure how much work it would be. also there is the workaround of sorting the input files in case writing to one of these is desired. It is an edge case. Maybe documenting it for now could be good enough?

Edit: In case this did not come through, I share @thaliaarchi s scepticism.

[nicowilliams]

Yes, this is a big deal. In general I expect tools that support -o FILE to also support using the same file as an input. Think of sed -i -- -i means "in-place". This is very tricky stuff to get right.

[wader]

@wader Any more thoughts on this?

Sorry i should i reviewed this earlier, looks overall good 👍 had some questions. Would be good if some more maintainer could have a look.

[wader]

what should happen if output file arg is used more than once?

[thaliaarchi]

The file from the first flag is opened, then closed when the second flag is parsed. I think this is fine.

[nicowilliams]

No. Each file will be truncated, and only the last one will be written to (after being truncated). This seems like a footgun.

[christf]

What should the user interface be in this case? Should jq -o a -o b . inputfile be allowed?
Writing multiple -o options feels odd to me.

In any case, it seems to me that the fopen/fclose bit needs to move to after all options have been parsed into the proximity of line 565 - because even if it should be allowed to specify -o only once, the code could catch a violation of that rule only after the first output file has already been truncated, which would potentially truncate inputs.
In case a temporary file is used as output to be renamed afterwards, not moving the fopen() till after parsing all options would leave temporary files in the file system, so I think the fopen needs to move in all cases and I had not considered that in my original commit.

[thaliaarchi]

I'll add a test for the same file used in --output-file and as an input, then I think this is ready. I suppose in combination with --from-file, --rawfile, and --slurpfile could also be tested.

[thaliaarchi]

I've added those additional tests.

[wader]

Looks good too me. Hope some more maintainer have thought about this

[nicowilliams]

Maybe these should always have gone to stderr though, no? I would be for making that change.

[nicowilliams]

What are the rename-into-place or truncate-and-rewrite semantics? I think that needs to be stated clearly, though it's true that many tools that have -o FILE options don't say so clearly.

[nicowilliams]

We had some lengthy discussion of -o FILE and -i (for "in-place", like sed's -i option) some years ago, maybe eight or ten years ago. I wrote https://github.com/nicowilliams/inplace as a side effect of that discussion.

Basically, there are footguns here, and not having a -o FILE or -i option seemed like the better option at the time.

Here's my take:

• sed -i creates a temp file and renames it into place; jq -o FILE should do the same -- this is a simple change, just create a string that is .<FILE>.tmp, fopen() it with "w" (truncate), then at the end rename() it into place

• jq -i is probably a lot of work to implement, unfortunately, and would probably be confusing, so jq -o FILE is best, but we should always be able to add -i later.

[nicowilliams]

@wader thanks for reviewing this and pinging me. @thaliaarchi I think this is close, but rename into place semantics will be much safer, and should be a simple change.

[christf]

We had some lengthy discussion of -o FILE and -i (for "in-place", like sed's -i option) some years ago, maybe eight or ten years ago. I wrote https://github.com/nicowilliams/inplace as a side effect of that discussion.

Basically, there are footguns here, and not having a -o FILE or -i option seemed like the better option at the time.

Here's my take:

* `sed -i` creates a temp file and renames it into place; `jq -o FILE` should do the same -- this is a simple change, just create a string that is `.<FILE>.tmp`, `fopen()` it with `"w"` (truncate), then at the end `rename()` it into place

* `jq -i` is probably a lot of work to implement, unfortunately, and would probably be confusing, so `jq -o FILE` is best, but we should always be able to add `-i` later.

using a temporary file has pitfalls too. This write-up is for java but the principle applies here too https://www.javathinking.com/blog/what-is-a-safe-way-to-create-a-temp-file-in-java/
It asks for

• names that cannot be guessed
• deletion on gracefule exit and application crashes

[nicowilliams]

using a temporary file has pitfalls too. This write-up is for java but the principle applies here too https://www.javathinking.com/blog/what-is-a-safe-way-to-create-a-temp-file-in-java/ It asks for

* names that cannot be guessed

* deletion on gracefule exit and application crashes

Since you're not trying to lock prior to truncation you just give up on "names that cannot be guessed" and then you don't have to worry about "deletion on gracefule exit and application crashes".

Using .<FILE>.tmp for this is perfectly reasonable and is superior to truncation.

[nicowilliams]

$ # what should happen here:
$ jq -f p -o a a
$ # ?
$ # Answer (a): a ends up empty
$ # Answer (b): a ends up having a transformation of its original content

$ # what should happen here:
$ jq -f p -o a a a
$ # ?
$ # Answer (a): a ends up empty
$ # Answer (b): a ends up having twice the transformation of its original content

$ # How is this:
$ jq -f p -o b a
$ # better than
$ jq -f p a > b
$ # ?
$ # Answer: well, if `set -o noclobber` were in effect then
$ #         this allows one to avoid having to unset it,
$ #         but then why set it?

On Linux strace -e trace=file sed -i 's/foo/foobar/' a shows:

openat(AT_FDCWD, "a", O_RDONLY)         = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=4, ...}, AT_EMPTY_PATH) = 0
openat(AT_FDCWD, "./sedSmiGtS", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=4, ...}, AT_EMPTY_PATH) = 0
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_EMPTY_PATH) = 0
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=4, ...}, AT_EMPTY_PATH) = 0
rename("./sedSmiGtS", "a")              = 0

but sed could just as well have used .a.tmp.

Yes, using .a.tmp would be racy, but no more so than truncating a without locking is anyways.

Notice that -o a is one more character to type than > a, so the only value I see in -o FILE is:

• familiarity -- for some reason many users know -o FILE better than > FILE
• if you're using set -o noclobber then -o FILE works around that, allowing you to not have to either unset it nor unlink FILE first

I guess the familiarity reason is the reason several PRs have been opened for this feature since the set -o noclobber reason is not mentioned either here nor in #3367. Given so much demand for -o FILE I suppose we should finally add it, but see the above questions.

Why does sed -i do what it does? This seems worthy of some research. It should shed light on what jq ought to do here.

[nicowilliams]

I forgot to ask earlier:

$ # What is the intent here:
$ jq -f p -o a -o b c
$ # ?
$ # Answer (a): `a` gets truncated, `b` gets truncated, `a` ends up
$ #              empty, `b` ends up with a transformation of `c`
$ # Answer (b): `a` and `b` end up with the same content (a
$ #              transformation of `c`)
$ # Answer (c): this should be an error (and `a` gets truncated)!
$ # Answer (d): this should be an error (and neither `a` nor `b` gets truncated)!
$ # Answer (e): something else?

Currently the answer would be (a). This is bound to be surprising to some users.

Though:

$ date >a >b
$ file a b
a: empty
b: ASCII text

but that brings me back to asking: how this is better than the shell's I/O redirection? And, ah, right: one way in which -o FILE is better than I/O redirection is that if you're using spawn()/execve()/posix_spawn() without using a shell but might still want to do output redirection, and it's easier to use -o FILE then than write the code for I/O redirection (it's just two more argument vector elements), but that thought brings me back to asking about motivation:

Why add -o FILE?

I'm not asking that to annoy you. I'm genuinely curious. My guess is: familiarity. Familiarity is a reasonable and sufficient reason for adding -o FILE! But I'm still curious what the answers should be to cases like jq -f p -o a a a.

[nicowilliams]

Also, as noted by @wader earlier, jaq has -i/--in-place:

But then I notice that it says singular "input" and "output": Overwrite input file with its output. But jq accepts multiple inputs, so one might expect -i/--in-place to mean "for each input", not "for the input". Then again, jaq doesn't say "the input" or "the output" either. And now I'm wondering what jaq does for -i/--in-place!

IMO we should prefer to add -i/--in-place like jaq did, though first we should investigate what jaq's exact semantics for -i/--in-place are. Though we might still add -o FILE, naturally.

[nicowilliams]

Ok, thinking about it, here's my current take:

• -o FILE and -i/--in-place are separable, so this PR should continue to be about -o FILE only even though (and hopefully a future PR should add -i/--in-place)
• -o FILE and -i/--in-place should have similar semantics when FILE is both an input and an output file
• -o FILE and -i/--in-place should rename a temp file into place, much like sed does
• any future -i/--in-place should do in-place (rename into place) rewriting of every input file, so jq -f p -i a b c should process a in-place, then b in-place, then c in-place
• multiple -o options should be an error
• no side-effects should happen in the multiple -o options error case

and:

• the temp file could be made with mkstemp() or similar (even though this requires more work for the WIN32 case), and I'm not concerned about garbage temp files piling up (jq shouldn't crash, and nothing should be killing jq)
• but the temp file could just be named .jq<FILE> or .jq<FILE>.tmp to avoid the risk of garbage temp files piling up, and I'm not concerned about racing anymore than this PR already is -- if you race multiple jq -o a ... processes then you will get surprised, though I wouldn't be averse to locking the temp file prior to truncating it (open, lock, check that it's still the temp file, truncate, process inputs and write outputs to the temp file, rename into place) to serialize concurrent jq processes that want to -o to the same file.

@thaliaarchi @christf @wader thoughts?

[nicowilliams]

FYI, doing some research on sed -i and perl -i, it seems that:

• GNU sed added the -i option and it always worked as rename-temp-file-into-place -- I don't think truncation was ever tried and failed
• perl -i opens the input file, unlinks it, creates the file again, and writes to it (on Windows this requires a backup suffix)
• WIN32 now has FILE_DISPOSITION_FLAG_POSIX_SEMANTICS to enable unlinking open files

[nicowilliams]

Using a well-known temp file naming scheme would be a problem if the file is in a world-writable directory. So it'd have to be mkstemp() if we go that route OR we'd have to open the temp file for writing, lock it, check that it's still the temp file, check that it's owned by the same user as getuid(), check that it wasn't a symlink, truncate it, etc.

[christf]

but that brings me back to asking: how this is better than the shell's I/O redirection? And, ah, right: one way in which -o FILE is better than I/O redirection is that if you're using spawn()/execve()/posix_spawn() without using a shell but might still want to do output redirection, and it's easier to use -o FILE then than write the code for I/O redirection (it's just two more argument vector elements), but that thought brings me back to asking about motivation:

Why add -o FILE?

I'm not asking that to annoy you. I'm genuinely curious. My guess is: familiarity. Familiarity is a reasonable and sufficient reason for adding -o FILE! But I'm still curious what the answers should be to cases like jq -f p -o a a a.

It is a great question and I am not at all annoyed by it. #3367 contains the gist "When calling jq inside a docker context this will be helpful as it means jq will not have to be called from within a shell with output redirection." but let me elaborate:

The motivation is to use the official jq docker image as defined in https://github.com/jqlang/jq/blob/master/Dockerfile, in a Tekton-pipeline (a continuous integration tool that runs on kubernetes. Its pipelines are described in steps that are executed in a container each. When there is data to be passed between steps, this happens via files in an input or output area). As best practice dictates, the jq image published by this project is minimal.

This means that in the tekton use-case, there is "just jq" and no shell available.
Adding a shell to the jq image is certainly not the right thing to do, so using ">" is not an option out of the box.
Maintaining another image that contains jq and a shell is my current workaround but it is clunky, increases overhead in maintenance and gives potential attackers a very useful tool (the shell).

And of course, familiarity is a factor: Many tools support -o, so it feels natural to me if jq does so too. I may be overlooking something but to me it looks the most elegant approach for this use case.

Edit: --in-place will not help in the above use case, as the input area and the output area are separated (they are separate workspaces/mounts).

[nicowilliams]

It is a great question and I am not at all annoyed by it. #3367 contains the gist "When calling jq inside a docker context this will be helpful as it means jq will not have to be called from within a shell with output redirection." but let me elaborate:

[...]

This means that in the tekton use-case, there is "just jq" and no shell available.

Got it. This is basically the shell-less spawn/exec argument I mentioned.

We should definitely add this then. To me this is a much stronger argument than the familiarity argument.

So now let's settle the semantics of -o FILE. I get that the current PR state is the absolute simplest semantics to implement portably, that using mkstemp() means reverting 77dcaf3. Or we could use a non-randomized temp file but taking care to implement it correctly as I mentioned above (especially if we add locking then we need even more portability code, though IMO we shouldn't do any locking). But I do think that jq -f p -o a a should behave in a non-surprising way, much like sed -i.

[christf]

my two cents:
$ jq -f p -o a a a should be b) because this is what happens without specifying -o. This is the case that seems to require a temporary file.
jq -f p -o a a should be b) but currently is a)

If there is a case that requires using temporary files (I remember acutely having said "yikes" to temp files above. I still do not like them. They increase memory usage in the general case just to allow implementing a corner case correctly and leaving them around after the application is killed, is a problem. I disagree that we can assert jq is never being killed. jq could be used to handle very large data for example and the OOMkiller might come in. I think jq should try to clean up in every case but SIGKILL, this means anothet increase of the code base.), we might as well lean on them to implement the feature consistently.

Would reverting 77dcaf3 be bad? It certainly was right to commit this at the time when it wasn't used.

[nicowilliams]

my two cents: $ jq -f p -o a a a should be b) because this is what happens without specifying -o. This is the case that seems to require a temporary file. jq -f p -o a a should be b) but currently is a)

Right!

Every way I can think of to avoid needing a temp file ends up with some obnoxious pathology. Imagine we tried to avoid a temp file when the -o FILE doesn't exist, so we could try open(fn, 'O_CREAT | O_EXCL | O_WRONLY', 0600) and then fdopen() that if it succeeds: well, consider this case jq -f p -o a b a! We'd process all of b, write it to a, then process a writing to a and... ooof, that'd be an infinite loop that consumes all available disk space! Say we tried to avoid a temp file when we first scan all the arguments to confirm that -o FILE does not refer to an input file: symlinks and relative paths and what not can make it impossible to tell -- think jq -f p -o a b ./a.

I believe we can't avoid a temp file. Like you I'd rather avoid that.

We could have -O FILE mean "truncate immediately" and -o FILE mean "like sed -i". This if you have a use case for it (which I believe you do).

[...]

Would reverting 77dcaf3 be bad? It certainly was right to commit this at the time when it wasn't used.

It's fine to revert it.

[nicowilliams]

Ok, so here's my proposal:

• -O FILE should work like -o FILE in this PR as of right now -- it can be useful if documented as "truncates immediately"
• -o FILE should first try open("<dirname(FILE)>.jq-<basename(FILE)>.tmp", O_CREAT | O_EXCL | O_WRONLY, 0600) and if EEXIST then use mkstemp() (reverted), and either way use a signal handler to unlink the temp file if killed with any death signal that can be caught that isn't ignored, then when done close all the input files and rename() it into place

EDIT: I've got Claude writing a commit for this. We'll see how it goes.

[nicowilliams]

@christf @thaliaarchi I've pushed two commits, 5be80fe which restores mkstemp() and 84fc28e that makes the -o you had be -O and adds a -o that tries to use a predictable temp file else falls back on mkstemp() and ultimately renames the file into place.

Please review!

[nicowilliams]

You can see why 13 years ago I didn't care for -o or -i. This is tricky stuff, and I don't recall anyone bringing up the no-shell use-case. Ah well.

[nicowilliams]

@thaliaarchi any particular reason this PR targets the 1.8.1 branch?

[thaliaarchi]

@thaliaarchi any particular reason this PR targets the 1.8.1 branch?

I'm not sure where you're seeing this. The merge base is master, though I haven't rebased in a while so it's not branched from HEAD.

[thaliaarchi]

I notice you kept the behavior of my version, renamed as -O/--clobber-output. With the temp-then-rename approach, this doesn't seem very useful to keep, and the claim you've documented of it being faster isn't very strong to me, as it's only a few extra fs operations. Do you have a use case in mind? It might be useful so another tool can read the output file while jq is writing, but redirection can do that and I doubt you'd have the Docker/other constraints and want to do this.

[christf]

What is the reason for keeping logic to have a predictable and an unpredictable name?

[christf]

I notice you kept the behavior of my version, renamed as -O/--clobber-output. With the temp-then-rename approach, this doesn't seem very useful to keep, and the claim you've documented of it being faster isn't very strong to me, as it's only a few extra fs operations. Do you have a use case in mind? It might be useful so another tool can read the output file while jq is writing, but redirection can do that and I doubt you'd have the Docker/other constraints and want to do this.

I am happy to test my use case with both and will come back.

[nicowilliams]

I notice you kept the behavior of my version, renamed as -O/--clobber-output. With the temp-then-rename approach, this doesn't seem very useful to keep, and the claim you've documented of it being faster isn't very strong to me, as it's only a few extra fs operations.

The "faster" claim is Claude's. Oops. But as to use-case and utility, -O lets you: a) not worry about temp file cleanup, b) tail -f the output file. Not sure that's strong enough to keep.

[nicowilliams]

What is the reason for keeping logic to have a predictable and an unpredictable name?

Just a thought I had, that whenever the O_EXCL case works then you don't have to worry about garbage temp files piling up in pathological cases. But yeah, sed -i doesn't care, so maybe we shouldn't either.

[christf]

I notice you kept the behavior of my version, renamed as -O/--clobber-output. With the temp-then-rename approach, this doesn't seem very useful to keep, and the claim you've documented of it being faster isn't very strong to me, as it's only a few extra fs operations. Do you have a use case in mind? It might be useful so another tool can read the output file while jq is writing, but redirection can do that and I doubt you'd have the Docker/other constraints and want to do this.

I am happy to test my use case with both and will come back.

both (-o and -O) work for my use case with tekton

[christf]

Should we delay the actual opening of the output file until after parsing all options to avoid truncating multiple files? It is a weird corner case but when it occurs, it almost certainly will be annoying for the user.

[thaliaarchi]

I'd rather remove -O/--clobber-output, keeping only temp-then-rename -o/--output-file.

[christf]

I'd rather remove -O/--clobber-output, keeping only temp-then-rename -o/--output-file.

I agree. Jq should decide for one of the implementations to keep code base maintainable. Since there are inherent issues with -O, choosing -o seems prudent.

[01mf02]

Hi everyone, jaq maintainer here. :)

Just to give a bit more context: --in-place in jaq is a bit tricky, in particular because of its interaction with --slurp. In a nutshell, this forces jaq to slurp files individually when multiple input files are given. So I think I would consider this as a quite different problem from -o.

I think that if -o / --output-file means "open a temporary file, write all output to that file, then move the temporary file to the given location", then I could implement this also quite easily in jaq, without creating entirely new code paths.