Description
Since 0.14.5, svgdigitizer.pdf.Pdf.build_identifier translates LaTeX escape sequences to unicode via a latin-1 round-trip:
entry.persons["author"][0].last_names[0].encode("latin-1").decode("latex+latin")
(and the same for the title field). This raises UnicodeEncodeError whenever the BibTeX entry already contains UTF-8 characters that are not representable in latin-1, e.g. ć (U+0107), ž (U+017E), or an em dash — (U+2014) in the title:
>>> from pybtex.database import parse_string
>>> from svgdigitizer.pdf import Pdf
>>> bib = parse_string(
... '@article{jovic_1996_test_1, author = {Jović, BM}, title = {Some title},'
... ' year = {1996}, pages = {1--10}, journal = {J}}',
... bib_format="bibtex",
... )
>>> Pdf.build_identifier(bib)
Traceback (most recent call last):
...
UnicodeEncodeError: 'latin-1' codec can't encode character 'ć' in position 4: ordinal not in range(256)
This breaks validate-bib-keys in echemdb/electrochemistry-data, where the bibliography deliberately stores authors such as Jović, Marinković, and Adžić as plain UTF-8 (LaTeX accent encodings are explicitly forbidden there by validate-bib-utf8).
Suggested fix
Only apply the latin-1/latex+latin round-trip when it is possible, and fall back to the string as-is (it is already unicode) otherwise, e.g.:
def _latex_to_unicode(value):
try:
return value.encode("latin-1").decode("latex+latin")
except UnicodeEncodeError:
return value # already plain unicode with non-latin-1 characters
Alternatively, encode with errors="backslashreplace"-style handling via latexcodec's own latex+latin1 codec.
Observed with svgdigitizer 0.14.5.
Description
Since 0.14.5,
svgdigitizer.pdf.Pdf.build_identifiertranslates LaTeX escape sequences to unicode via a latin-1 round-trip:(and the same for the title field). This raises
UnicodeEncodeErrorwhenever the BibTeX entry already contains UTF-8 characters that are not representable in latin-1, e.g.ć(U+0107),ž(U+017E), or an em dash—(U+2014) in the title:This breaks
validate-bib-keysin echemdb/electrochemistry-data, where the bibliography deliberately stores authors such asJović,Marinković, andAdžićas plain UTF-8 (LaTeX accent encodings are explicitly forbidden there byvalidate-bib-utf8).Suggested fix
Only apply the latin-1/
latex+latinround-trip when it is possible, and fall back to the string as-is (it is already unicode) otherwise, e.g.:Alternatively, encode with
errors="backslashreplace"-style handling via latexcodec's ownlatex+latin1codec.Observed with svgdigitizer 0.14.5.