Change max tokens in python

[rosmur]

Any documentation available on how to increase the max tokens? Its defaulting to 100 seemingly.

[Blaizzy]

Just pass --max-tokens

[rosmur]

Thanks. Is there a reason the default is such a small number?

[Blaizzy]

Not really.

I think when I initially built this package I didn't want it to generate a large blob of text when I was testing.

I can increase it. How much would you like to be the default?

[rosmur]

thanks, I think 2048 or 4096 would be good. Those are typical settings I've encountered for generate functions in other packages

[Blaizzy]

that's too high

Most models don't output that much text.

How about 1000?

[rosmur]

yeah, I think thats ok. I've seen minimum context length defaults of 2k but those are for chat completion clients where naturally the context length increases with chat turns, but since this is a generate function 1000 makes sense