Bff/token optimisation - #268
Conversation
Bff training tweaks
fixed task name
|
This is very useful! One small findings: |
|
Ah yes, thanks for catching that - removed the file. I should have put some context in the original PR. This is to fix the issue I flagged here #266 |
|
@bugface – let me know if you're happy to merge :) |
I think it looks great. |
|
I think we're good to merge then, @bugface :) |
|
news about this? @bugface |
matteocacciola
left a comment
matteocacciola
left a comment
There was a problem hiding this comment.
If the default value of replace_additional_special_tokens is False, why do you assign False value when you explicit its usage?
|
Ah yeah, I guess bad style. At the time I wanted to be explicit because this bug was so opaque. Are you happy to merge if I fix that? |
|
I could be interested in patching it into my fork, if you fix and explain the changes. Thank you.
|
| if len(set(list_of_tokens) - set(self.tokenizer.all_special_tokens)) > 0: | ||
| newly_added_num = self.tokenizer.add_special_tokens({"additional_special_tokens": sorted(set(list_of_tokens))}, replace_additional_special_tokens=replace_additional_special_tokens) | ||
| else: | ||
| newly_added_num = 0 |
There was a problem hiding this comment.
I guess this is the real solution to the detected bug, isn't it?
4 participants
PR to fix: #266