Is your feature request related to a problem? Please describe.
I find hard to book-keep all the KV cache updates and backtracks between the draft_subnet and verify_subnet. Need a consistent way to perform these actions; especially when we have batches where an uneven number of draft tokens are accepted for each prompt.
Describe the solution you'd like
A SpeculativeCacheManager that will abstract away all of the KV Cache updates and backtracks.
Describe alternatives you've considered
Making classmethods in the CleavedAutoModelForCausalLM class; this seemed to violate the single responsibility principle and made the class very verbose.
Additional context
None atm.
Is your feature request related to a problem? Please describe.
I find hard to book-keep all the KV cache updates and backtracks between the
draft_subnetandverify_subnet. Need a consistent way to perform these actions; especially when we have batches where an uneven number of draft tokens are accepted for each prompt.Describe the solution you'd like
A
SpeculativeCacheManagerthat will abstract away all of the KV Cache updates and backtracks.Describe alternatives you've considered
Making classmethods in the
CleavedAutoModelForCausalLMclass; this seemed to violate the single responsibility principle and made the class very verbose.Additional context
None atm.