Sharding Prototype II: implementation on Array

[jstriebel]

This PR is for an early prototype of sharding support, as described in the corresponding issue #877, an alternative to it's cousin PR #876. Please also see the discussion there. It serves mainly to discuss the overall implementation approach for sharding. This PR is not (yet) meant to be merged.

This prototype

• allows to specify shards as the number of chunks that should be contained in a shard (e.g. using arr.zeros((20, 3), chunks=(3, 3), shards=(2, 2), …)).
  One shard corresponds to one storage key, but can contain multiple chunks:
  
• ensures that this setting is persisted in the .zarray config and loaded when opening an array again, adding two entries:
  • "shard_format": "indexed" specifies the binary format of the shards and allows to extend sharding with other formats later
  • "shards": [2, 2] specifies how many chunks are contained in a shard,
• adds a small script sharding_test.py for demonstration purposes, this is not meant to be merged but servers to illustrate the changes.

---

The currently implemented file format is still up for discussion. It implements "Format 2" @jbms describes in #876 (comment).

Chunks are written successively in a shard (unused space between them is allowed), followed by an index referencing them.
The index holding an offset, length pair of little-endian uint64 per chunk, the chunks-order in the index is row-major (C) order,
e.g. for (2, 2) chunks per shard an index would look like:

| chunk (0, 0)    | chunk (0, 1)    | chunk (1, 0)    | chunk (1, 1)    |
| offset | length | offset | length | offset | length | offset | length |
| uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 |

Empty chunks are denoted by setting both offset and length to 2^64 - 1. All the index always has the full shape of all possible chunks per shard, even if they are outside of the array size.

For the default order of the actual chunk-content in a shard I'd propose to use Morton order, but this can easily be changed and customized, since any order can be read.

[mkitti]

Cross posting from #876 (comment). Perhaps nbytes is a less ambiguous term than length to describe the number of bytes in a chunk?

[jstriebel]

A similar proposal is now formalized as a Zarr Enhancement Proposal, ZEP 2. The basic sharding concept and the binary representation is the same, however it is formalized as a storage transformer for Zarr v3.

Further feedback is very welcome! Either

• on the abstract specification as comments on PR Review of the ZEP2 spec - Sharding storage transformer zarr-specs#152, or
• on the Python implementation PR Sharding storage transformer for v3 #1111.

I'll close this PR to consolidate the open sharding threads, please feel free to continue any discussions or add feedback on the PRs mentioned above.