sync: whole prefixes are silently dropped when the object store does not list keys in lexicographic order (e.g. TOS buckets with hierarchical namespace)

[SecretSun]

What happened:

Syncing from a Volcengine TOS bucket (hierarchical namespace enabled, accessed through its S3-compatible endpoint) with juicefs sync --list-threads 2 --list-depth 2 --enable-checkpoint, the final summary reported only a small fraction of the bucket:

Found: 99236, excluded: 0 (0 B), skipped: 5 (0 B), copied: 99229 (2.83 TiB), extra: 0 (0 B), failed: 226
FATAL: failed to handle 226 objects

while the bucket holds ~83 TiB. The 226 "failed" entries are not objects but prefixes whose listing aborted. None of the objects below them are counted in Found, so the shortfall is invisible unless you compare with the console.

The -v log shows two error shapes:

<ERROR>: The keys are out of order: marker "", last "a/b/dir2/" current "a/b/.cache/huggingface/" [listAll@sync.go:290]
<ERROR>: Failed to restart producer for prefix a/b/: list s3://...: found invalid key a/b/pkg-1.0/ from List, prefix: a/b/, marker: a/b/zzz/ [restorePrefixFromCheckpoint@sync.go:1814]

and with delimiter listing on a directory with more than 1000 entries:

<ERROR>: list prefix a/pkgs/: list s3://... with delimiter: found invalid key a/pkgs/libwebp-1.2.0-h89dd481_0.conda from List, prefix: a/pkgs/, marker: a/pkgs/libwebp-1.2.0-h89dd481_0/ [startProducer@sync.go:1988]

In --dry mode the same listing error crashes the process instead:

panic: runtime error: invalid memory address or nil pointer dereference
github.com/juicedata/juicefs/pkg/sync.startProducer.func2.1  pkg/sync/sync.go:1989

What you expected to happen:

Either the sync lists and copies everything, or it fails loudly with a count that reflects the objects that were not synced. A dry run should report listing errors, not panic.

How to reproduce it:

1. A TOS bucket with hierarchical namespace, containing a directory whose entries include both x/ and x.<ext> (e.g. a conda package cache), more than 1000 entries in that directory, and some nested directories.
2. juicefs sync --dry s3://bucket.tos-s3-<region>.ivolces.com/prefix/ <dst> --list-threads 1 → The keys are out of order immediately.
3. juicefs sync --dry ... --list-threads 8 --list-depth 64 on the >1000-entry directory → found invalid key, then the nil pointer panic in dry mode.

Root cause:

TOS (HNS) returns ListObjectsV2 results in directory order: entries of a directory come before its sub-directories, and directory names are ordered without the trailing /, so foo/ comes before foo.conda. juicefs sync assumes strict lexicographic order in three places:

1. pkg/object/s3.go / tos.go List() rejects any key below StartAfter with found invalid key, even on continued pages where the server follows its own continuation token. listCommonPrefix passes the last key of the client-sorted page as marker, so a directory with >1000 entries fails as soon as a page boundary falls between x/ and x.<ext>.
2. listCommonPrefix appends pages in server order; the per-page sort in the backend does not order entries across pages, and produce() merges src and dst by key.
3. failed is only created when !config.Dry, but listing failures call failed.Increment() unconditionally.

Additionally, with --enable-checkpoint, a prefix that fell back to flat listing is stored with list_depth: 0 and the checkpoint key hash does not include --list-depth, so re-running with a larger depth restores the old depth and fails again until --checkpoint-force-reset is used.

Environment:

• JuiceFS version: juicefs version 1.4.1+2026-07-30.0b90c7d; the code paths are unchanged on current main.
• OS: Ubuntu 24.04, x86_64.
• Source: Volcengine TOS, hierarchical namespace bucket, S3-compatible endpoint (JFS_S3_VHOST_STYLE=1). Destination: Tencent COS.

I have a fix for the three points above and will open a PR referencing this issue.