What is the purpose of positional offset in the rotary positional embedding implementation?

[ifed-ucsd]

I was looking at the rotational position embedding code path (

recurrent-memory-transformer-pytorch/recurrent_memory_transformer_pytorch/recurrent_memory_transformer.py

Line 414 in 35cd18d

mem_rel_dist = 10000

) and noticed this comment:

rotary embedding - offset main positions by 10000, and keep all memories at position 0

What is the intuition for this design choice? I don't see this detail anywhere in the RMT papers; did I miss something? Do you have a reference that does this kind of offsetting?

Thanks!