Input embedding layers, paired with the corresponding block type.
Registry: praxis.EMBEDDING_REGISTRY (15 entries)
Value: functools.partial(<function praxis.embeddings._compose>, [('tok', {})])
Value: functools.partial(<function praxis.embeddings._compose>, [('tok', {}), ('hash', {'group_sizes': [3, 4, 5], 'functions': 1})])
Value: functools.partial(<function praxis.embeddings._compose>, [('tok', {}), ('hash', {'group_sizes': [3, 4, 5], 'functions': 4})])
An embeddings module with optional projection layer and dropout. If embed_size differs from hidden_size, a linear projection layer is added to map the embeddings to the required hidden dimension.
Source: praxis/embeddings/projected.py:11
N-gram hash embedding: sums table lookups over byte windows of several sizes and hash functions, computing vectors from byte n-grams rather than retrieving a per-token row. Has no single tie-able table by design.
Collision resistance comes from functions, not from hash_vocab. A single hash
maps every distinct n-gram onto exactly one row, so two n-grams sharing a bucket become
the same vector and are unrecoverable downstream; the only defence is a wider table, and
byte n-grams outnumber ...
Source: praxis/embeddings/hash.py:41
Praxis embeddings with learned positional encodings (GPT2-style). Uses Sequential organization of layers.
Source: praxis/embeddings/positional.py:11
Per-byte token table for byte-latent encoders; the weight-tying target.
Sizes itself from the encoder's declared input layout (input_dim /
input_vocab_size) when present, else from the global config.
Source: praxis/embeddings/byte.py:4