zstream: Add subcommand applying zvol stream to block device or raw disk image

[ryan-moeller]

Motivation and Context

There are situations where it is desirable to migrate a workload from a zvol to another format, such as a real device or a disk image file. For an offline migration, dd is adequate. When the zvol must remain in use during the operation, a snapshot can provide a consistent, stable state to copy, but alone this will still lose any changes that occur after the snapshot.

ZFS incremental replication offers a clean solution by capturing the bulk of the transfer in an initial snapshot, with a subsequent incremental snapshot to capture only the changes after the initial snapshot (thus completing much more quickly). The missing piece for this operation is the ability to apply zvol send streams to the new medium.

Description

Introduce a zstream raw subcommand that applies the operations in a zvol send stream to a raw device or regular file. The main stream operations of interest are WRITE records which cause data to be written to the target volume and FREE records which cause data to be discarded from the target volume.

For now, writes are applied using pwrite(2) (after decompression if applicable), and for generality frees are applied by writing zeros from a single zeroed page using pwritev(2). Zvols are initialized with a large free to the end (-1) in the send stream, which is translated to a series of ftruncate(2) operations for regular files or simply skipped for devices.

Future work is underway for optimizing writes by combining contiguous writes into a single pwritev(2) call and frees by issuing BLKDISCARD (on Linux). Other FREE optimizations such as hole punching for files or TRIM/UNMAP mechanisms for FreeBSD are not planned, but these would be welcome contributions.

Support for the following stream features is implemented:

• DMU_BACKUP_FEATURE_EMBED_DATA
• DMU_BACKUP_FEATURE_LZ4
• DMU_BACKUP_FEATURE_LARGE_BLOCKS
• DMU_BACKUP_FEATURE_COMPRESSED
• DMU_BACKUP_FEATURE_ZSTD

It is possible to apply streams repeatedly or even in the wrong order, as no ZFS/stream metadata is preserved by the target volume. To offer some safety mechanism, the zstream raw operation prints out the ending snapshot guid at the completion of the stream operations. This can then optionally be provided to future invocations as zstream raw -g GUID to sanity check the fromguid before applying the subsequent stream, aborting on mismatch. Handling of this GUID is left as the responsibility of the user.

How Has This Been Tested?

A new ZTS test is added which does some basic verification of a few operations using dd and xxhash. Separate manual testing with XFS and ext4 was done but is not automated for the test suite. Exhaustive testing of every combination of features would add excessive time to test runs.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[ryan-moeller]

• Rebase
• Skip writing zeros on free to end (-1 length)
• Init/fini libzpool subsystems similar to zstream recompress

There is work underway to implement a few optimizations as mentioned in the description, but I'm opening this for review now as those optimizations may be incorporated later or separately.

[ryan-moeller]

• Add a missing space in zstream.8

[ryan-moeller]

• Rebase
• Fix cstyle whitespace
• Allocate zeroes iovec array once with full size instead of growing with realloc on demand (it's not that big)

[behlendorf]

@ryan-moeller this will definitely be handy! After going through #18509 I felt the best way forward was to get that major refactoring merged first. It's now in master so when you get a chance please go ahead and rebase this, update it to use the new APIs and then I can take another look.

[GarthSnyder]

Now that #18509 is in, would you like me to take a shot at porting this to the chain system? It's likely to expose a few weaknesses of #18509 since it will be the first addition that wasn't explicitly contemplated in the original design.

[ryan-moeller]

Now that #18509 is in, would you like me to take a shot at porting this to the chain system? It's likely to expose a few weaknesses of #18509 since it will be the first addition that wasn't explicitly contemplated in the original design.

That would be great, thank you!

[GarthSnyder]

Excellent, give me a few days on it and I'll report back.

[GarthSnyder]

OK, back atcha. I submitted this as a PR against KlaraSystems/zstream-raw. There are more details in the PR, but it went pretty smoothly. It did motivate a couple of changes to the infrastructure, so thanks for the opportunity to look this over.

[ryan-moeller]

• Rebased
• Relocated test to the zstream group

I have not fully incorporated the changes @GarthSnyder proposed yet, this push was mainly to clean up his PR against my branch to simplify review there.

[ryan-moeller]

• Rebased
• Added a commit up front encompassing some refactoring from zstream: add "drop records" chain module #18755 and zstream: open up dump API to non-dump subcommands #18756 (will be rebased out before merge)
• Integrate with the new zstream chain processing system
• Dropped the safe_realloc commit which is no longer needed
• Tweaked manpage wording and example device name
• Annotate zstream_usage() as noreturn
• Renamed extend() helper to resize() since it is used to both shrink and extend a raw image file

[ryan-moeller]

• Rebased
• Fixed memory leak introduced in embedded write decompression after refactoring

[ryan-moeller]

• Rebase
• Optionally buffer and coalesce contiguous writes with pwritev(2) (-b # limits how many blocks can be buffered, default 1)
• Try using BLKDISCARD on Linux or SIOCGDELETE on FreeBSD to issue TRIM/UNMAP to block devices for FREE records instead of writing zeros
• Try using fallocate(2) on Linux or fspacectl(2) on FreeBSD to punch holes in files for FREE records instead of writing zeros
• Make randfree_file work for use in tests and enable compress_004_pos on FreeBSD now that it is available
• Update the ZTS test to create and exercise a filesystem on the zvol for extra coverage of WRITE and FREE record handling

This is all the additional features, so barring any minor corrections this should be ready to go.

[GarthSnyder]

My strong preference is that zstream commands should not accidentally run systems out of memory by default.

That would be my preference as well. 🙂 Even 4GB is a stretch, at least for this operation.

Here's what's going on. MAX_BATCH is just the granularity at which worker threads pick records out of the work queue, so it doesn't affect memory consumption. Records stay in situ within the queue while they're being worked on (checksummed, decompressed, etc.).

But the queue itself has a much larger capacity than MAX_BATCH, especially the checksum queues, since as (normally) first and last queues they also function as I/O buffers. The default is 1024 slots on input. You have a decompression step as well, which adds another 256 slots. So, the sliding window in memory can potentially be as long as 1280 records at any given moment. If you #define MONITOR_QUEUES, you can see the actual queue depths in real time. (Let me double-check that that's still working...yes, it is. That should probably be a secret command-line option.)

Were they all WRITEs, that'd be over 20GB of memory. But since most records are not WRITEs, you're seeing a baseline of 4GB. When you start adding buffers to the IOV queue, you're cherry-picking the WRITE records from the stream, so that memory adds up quickly. But 16 or 32 IOVs seems like a reasonable default. I'm not sure why you're seeing the explosion from 4GB to 12GB with a value of 32. That doesn't seem like it should be possible.

A default > 1 probably wouldn't have any effect on performance with these large WRITE sizes, but for smaller or fragmented WRITEs it might be of some benefit, and it makes -b less of an edge case.

That serial_drop_records() step in front of decompression is having a similar cherry-picking effect, since there are fewer small padding records in the decompression queue. Great for the decompressor! But not for memory. You might try moving that step behind decompression and see how that affects your RSS.

You can shrink the input queue by using the STANDARD_INPUT_STACK_Q macro, which adds a size parameter to STANDARD_INPUT_STACK. A smaller input queue can limit the chain's ability to saturate expensive downstream steps, which is why the default is high. But since you're only doing decompression and are relatively I/O-bound, you can probably use a much smaller queue.

[ryan-moeller]

Were they all WRITEs, that'd be over 20GB of memory. But since most records are not WRITEs...

For zvols, sends are by far mostly WRITEs to one object, and occasional FREEs:

SUMMARY:
        Total DRR_BEGIN records = 1 (0 bytes)
        Total DRR_END records = 1 (0 bytes)
        Total DRR_OBJECT records = 2 (0 bytes)
        Total DRR_FREEOBJECTS records = 1 (0 bytes)
        Total DRR_WRITE records = 4304609 (70131052544 bytes)
        Total DRR_WRITE_BYREF records = 0 (0 bytes)
        Total DRR_WRITE_EMBEDDED records = 645 (55664 bytes)
        Total DRR_FREE records = 2202 (0 bytes)
        Total DRR_SPILL records = 0 (0 bytes)
        Total DRR_OBJECT_RANGE records = 0 (0 bytes)
        Total DRR_REDACT records = 0 (0 bytes)
        Total records = 4307461
        Total payload size = 70131108208 (0x105423c970)
        Total header overhead = 1343927832 (0x501ab618)
        Total stream length = 71475036040 (0x10a43e7f88)

Unfortunately, I'm out of time for more work on this project outside of small last minute corrections. I can't justify additional budget for tuning the memory use of the chain mechanism. I hope the buffer limit is reasonable enough for this feature to be acceptable after raising the default to 32, and further improvements can be made separately.

[ryan-moeller]

• Change default buffer limit to 32
• Adjust manpage and test accordingly

[GarthSnyder]

Memory consumption is probably better addressed at a higher level anyway, e.g. a limit on the total sum of bytes in payloads that are allowed to be in flight at any given moment. Multiple subcommands rewrite payloads, so there probably needs to be a dp_original_size field just for bookkeeping.

[ryan-moeller]

• Rebase
• Fix whitespace

[behlendorf]

Yeah, we can tackle the memory consumption issues as a separate change. Thanks for working on this.

[behlendorf]

Merged. Thanks for working on this.