NEWS

up to: 6ddd7dd0a18b88bb0df2a4ac64f1b2fc9f856f40

Snowball 3.2.0 (UNRELEASED)
===========================

Compiler changes
----------------

* Bug fixes:

  - Fix detection of read errors for piped input which were effectively being
    ignored.  Bug introduced in 3.1.0.

  - The coverage counts for among functions were one too low.

  - Drop use of `restrict` in compiler source as it seems it isn't supported by
    MSVC 2019 and earlier.

* Compiler command-line options:

  - Change how -coverage reports among functions
    If the string matched we used to always report a string match, and
    additionally either "func-f" or "func-t" depending what the function
    signalled.
    Now we only report one thing - if the string matches but the function
    signals f we report "func-f"; if the string matches and the function
    signals t we reprot a string match.
    This gives a report that seems clearer, and requires logging and
    collating less data.

* Diagnostics:

  - Report an "info" diagnostic when we don't localise a variable into a
    recursive routine.

  - Fix spurious extra output of variable name on a line by itself after
    an "info" diagnostic saying we couldn't localise it.

  - Routine "not reachable" warnings are now reported at the definition which
    seems a better location than the declaration.

* Optimisations:

  - Grow large sumbol strings exponentially, like we already do for byte
    strings.

  - Eliminate some unnecessary copying when growing strings during compilation.

* Code quality:

  - Fix compiler memory leak when an unreachable routine contains an `among`.

* Other changes:

Generic code generation changes
-------------------------------

* Bug fixes:

* Optimisations:

  - Improve variable localisation machinery, and fix some obscure cases
    it was getting wrong.  All variables which are only used in one function
    after inlining in the shipped stemmers are now successfully localised.

  - Extend inlining to all calls to routines which consists of a single
    non-compound Snowball command.  The target language code generated for
    such commands should be comparable in size to the target language code
    to call a routine, and inlining may enable other optimisations (variable
    localisation and folding `not` into integer tests, for example).

  - Inline any non-recursive routine which is only used once (where that use is
    not as an among function).  When there's only one use this should if
    anything reduce the code size, and for target languages which don't
    inline themselves it eliminates the overhead of a call.  It may also enable
    more variables to be localised.

  - Normalise comparison with an integer literal so we merge equivalent among
    actions such as this example from arabic.sbl:

      [substring] among (
         // [...]
         '{a}{n}' '{w}{n}' '{y}{n}' ($(len > 5) delete) // present
         '{t}{m}{a}' ($(len >= 6) delete)
      )

* Code quality:

  - The Snowball manual says:

      For any string, the slice ends should be assumed to be unset until
      they are set with the two commands `[` and `]`.
  
    For many target languages, we actually initialise the slice ends
    (either because the target language requires doing so, or because the
    author of that generator thought it cleaner to do so, or perhaps just
    followed an existing target language generator which did this).

    In these situations, what the slice is set to does not matter since the
    Snowball language doesn't define values, so we now simply set both the
    slice start and end to 0 which is slightly simpler than the previous
    setting of the slice end to the length of the string.

* Documentation:

* Other changes:

Ada
---

* Bug fixes:

  - Generate valid Ada code for a Snowball routine/variable name which contains
    `__`.

* Optimisations:

* Code quality:

  - Use consistent case for `Debug()` helper function name.

* Documentation:

* Other changes:

C
-

* Bug fixes:

* Optimisations:

  - We now statically check if errors from a command are possible, and if not
    avoid generating extra code to propagate errors.

  - Reduce the work done when creating a stemmer.  We now memset() the whole
    allocation to zero which also takes care of initialising non-localised
    integer and boolean variables to 0.  If there's only one string variable
    we now skip pre-initialising it to NULL before we try to allocate it (this
    preallocation is done to facilitate cleaning up if we fail after partially
    allocating, but with only one string variable this can't happen).  We could
    skip this entirely if we know the implementation uses all-zero-bits in
    memory representation for NULL pointers, which all modern platforms seem to
    but the C standard doesn't guarantee.  However none of the current stemmers
    use more than one string variable, so we now always avoid the overhead in
    practice.

  - If an algorithm doesn't use strings then its close_env() only needs to call
    `SN_delete_env()`.  Both take the same parameters and return void, so we
    now `#define` the close_env function to `SN_delete_env` in the header which
    reduces code size and eliminates a small runtime overhead for closing a
    stemmer.  35 of the 38 stemmers we currently ship benefit from this.

  - Use a more sensible initial string size.
    We were allocating only enough space for a single character.  That
    won't be enough for words being stemmed.  It might be enough for
    some string variables, but we actually allocate an extra character to
    allow in-place zero termination as well as space for two ints to store
    the size and capacity, so the minimum total allocation is actually
    10 bytes when sizeof(symbol)==1 anyway.

  - Use putc() in debug() helper to write single characters.

* Code quality:

  - Limit generated internal C identifier lengths.  C90 only guarantees 31
    significant initial characters in an internal identifier.  C99 raised this
    to 63, and modern implementations are likely to have a high limit or not
    impose one, but it is easy to instead generate a short identifier using a
    number when the Snowball name is long.

  - The code now compile cleanly with MSVC with warning level `-W3` (which is
    apparently the default warning level when using the IDE).

* Documentation:

* Other changes:

C++
---

* Bug fixes:

  - Generate valid C++ code for a Snowball routine/variable name which contains
    `__`.

* Optimisations:

* Code quality:

  - Fix location of comment which should be at the start of the generated C++
    stemmer source but was actually after all the generated header includes.

* Documentation:

  - Remove -c from C++ stemwords help.  We don't support specifying the
    character set in the C++ version.

* Other changes:

  - Support generating wide-character stemmers which take and return
    std::wstring and work in wchar_t.  To use:

    make SNOWBALL_WIDE=1 clean
    make SNOWBALL_WIDE=1 cxx
    make SNOWBALL_WIDE=1 check_cxx

    See #267

C/C++
-----

* Bug fixes:

* Optimisations:

  - Snowball `among` is now implemented as a state-machine.  This gives a
    better theoretical performance (the new implementation is O(1) in the
    number of cases instead of O(log(n)), and is also faster for the current
    stemmers in benchmarks.

    The generated among tables are smaller than those for the existing
    implementation.

    The generated among tables are now just arrays of unsigned short,
    so require no dynamic load-time relocations if Snowball-generated C/C++
    code is included in a shared library.

    Among gating functions are now called by generated code, which
    means the C/C++ compiler can inline them and optimise better.

    The Snowball generated code for non-widechars now needs to know the
    endianness of the platform it is being built on.  The runtime code
    includes checks that should automatically detect this for GCC, clang and
    MSVC.  If the code can't determine the endianness it will #error at
    C/C++ compile-time.  In this case, if your platform has the POSIX
    endian.h header then you can define HAVE_ENDIAN_H.  Otherwise you'll
    need to define either SNOWBALL_BIGENDIAN or SNOWBALL_LITTLEENDIAN.

  - We now merge Snowball string literals which are identical or which are the
    prefix of another literal.  Partly addresses #262

* Code quality:

* Documentation:

* Other changes:

  - Flush stdout after each debug() call.  The debug command (`?`) is not a
    performance sensitive, and it can be confusing if its output is delayed by
    buffering.

C#
--

* Bug fixes:

* Optimisations:

* Code quality:

* Documentation:

* Other changes:

  - Support Snowball debug command (`?`).

Dart
----

* Bug fixes:

* Optimisations:

* Code quality:

* Documentation:

* Other changes:

  - Support Snowball debug command (`?`).

Go
--

* Bug fixes:

* Optimisations:

* Code quality:

* Documentation:

* Other changes:

  - Properly implement Snowball debug command (`?`).

Java
----

* Bug fixes:

* Optimisations:

* Code quality:

* Documentation:

* Other changes:

  - Support Snowball debug command (`?`).

Javascript
----------

* Bug fixes:

* Optimisations:

* Code quality:

  - Improve JSDoc TypeScript annotations.
    JS: Fix typescript annotations in generated code
    We need to pass `--check-js` to `deno check` to actually check .js
    files, which we now do (so we'll spot regressions).
    Closes #300, reported by alex-kinokon.
    stemwords.js: Improve typescript annotations
    Fix incorrect/missing annotations and adjust code so that type checking
    passes.
    Partially based on changes from alex-kinokon in #300.

  - Replace most @ts-expect-error annotations with JSDoc cast annotations.  A
    cast is better as it is more targetted, and also more self-documenting.

* Documentation:

* Other changes:

  - Support Snowball debug command (`?`).

Pascal
------

* Bug fixes:

* Optimisations:

* Code quality:

* Documentation:

* Other changes:

  - Support Snowball debug command (`?`).

PHP
---

* Bug fixes:

* Optimisations:

* Code quality:

  - Emit minimum length Unicode escapes.

* Documentation:

* Other changes:

  - Support Snowball debug command (`?`).

Python
------

* Bug fixes:

* Optimisations:

* Code quality:

* Documentation:

  - libstemmer_python_README: Fix typo.

* Other changes:

  - Include classifier for pt_BR since we document that our Portuguese stemmer
    "aims to work with both European Portuguese and Brazilian Portuguese".

  - Support Snowball debug command (`?`).

Rust
----

* Bug fixes:

* Optimisations:

* Code quality:

  - Emit minimum length Unicode escapes.

* Documentation:

* Other changes:

  - Support Snowball debug command (`?`).

Zig
---

* Bug fixes:

* Optimisations:

* Code quality:

* Documentation:

* Other changes:

  - Properly implement Snowball debug command (`?`).

New Code Generators
-------------------

Snowball Language Changes
-------------------------

* Handle `<-` in an arithmetic expression more gracefully.  In this context,
  the symbol `<-` isn't meaningful, so we now interpret it as `<` followed by
  `-`.  Examples such as `$x<-1` now parse the same way as `$x < -1` instead
  of giving a syntax error.

New stemming algorithms
-----------------------

* Add stemmer for Early Modern English
    This is a variant of the English stemmer which was previously only
    described on the website, but seems sufficiently useful to include in
    the distribution.

    A few additional rules have been added:

    + Remove th'-, t'-
    + Replace o'vr- with over-
    + Replace -e'er with -ever
    + Remove -edst (suggested by E. Timothy Uy in 2012!)
    + Remove -'d, -'dly, -'dst, -'st, -'t
    + Replace -'n, -'nd with -en
    + Replace -'r, -'rous with -er
    + Replace -'ri with -eri
    + Replace -'li with -ili
    + Replace -lie with -ly
    earlymodernenglish: Normalise æ and œ and long-s
    These ligatures appear in texts from this era significantly more than
    in modern English.  `œ` is not in iso-8859-1 so drop that.
    Fixes #261
    earlymodernenglish: Handle more prefixes/suffixes

Behavioural changes to existing algorithms
------------------------------------------

* Arabic:

* Armenian:

* Basque:

* Catalan:

* Czech:

* Danish:

* Dutch:

* English:

* Early Modern English:

* Esperanto:

* Estonian:

* Finnish:

* French:

  - Remove z'- which is a colloquial, nonstandard elision (for "vous"), and
    also a prefix used to represent a nonstandard liaison within a plural noun
    phrase, or eye dialect for a standard liaison.
    https://en.wiktionary.org/wiki/z%27#French

* German:

* Greek:

* Hindi:

* Hungarian:

* Indonesian:

* Irish:

* Italian:

* Lithuanian:

* Nepali:

* Norwegian:

* Persian:

* Polish:

* Romanian:

* Russian:

* Serbian:

* Sesotho:

* Spanish:

* Swedish:

* Tamil:

* Turkish:

* Yiddish:

Optimisations to existing algorithms
------------------------------------

* Arabic:

* Armenian:

* Basque:

* Catalan:

* Czech:

* Danish:

* Dutch:

* English:

* Early Modern English:

* Esperanto:

* Estonian:

* Finnish:

* French:

* German:

* Greek:

* Hindi:

* Hungarian:

* Indonesian:

* Irish:

* Italian:

* Lithuanian:

* Nepali:

* Norwegian:

* Persian:

* Polish:

* Romanian:

* Russian:

* Serbian:

* Sesotho:

* Spanish:

* Swedish:

* Tamil:

* Turkish:

* Yiddish:

Code clarity improvements to existing algorithms
------------------------------------------------

* Arabic:

* Armenian:

* Basque:

* Catalan:

* Czech:

* Danish:

* Dutch:

* English:

* Early Modern English:

* Esperanto:

* Estonian:

* Finnish:

* French:

* German:

* Greek:

* Hindi:

* Hungarian:

* Indonesian:

* Irish:

* Italian:

* Lithuanian:

* Nepali:

* Norwegian:

* Persian:

* Polish:

* Romanian:

* Russian:

* Serbian:

* Sesotho:

* Spanish:

* Swedish:

* Tamil:

* Turkish:

* Yiddish:

Build system
------------

* `make clean_runtime_tests` now first runs `make clean` for smoother switching
  to runtime test mode.

* The build system now supports MSVC better.  If CC matches `*cl` then it uses
  Microsoft's non-standard `-Fo:` and `-Fe:` instead of `-o`, which avoids lots
  of deprecation warnings.  Extensions `.o` and `.a` are still used (which
  triggers warnings from MSVC).  The command `ar` is used to create a static
  library so you'll also need the mingw toolchain installed for that.

* C++: Add missing dependency for runtime code.

* C/C++: Rename utilities.{c,cxx} to snowball_runtime.{c,cxx} to matches the
  header snowball_runtime.h which contains the prototypes for the external
  functions here.

* modules.txt: Fix alphabetical ordering of entries.

libstemmer
----------

* Fix incorrect ISO 639-2/B code "pers" for Persian.  This should be "per"
  (these codes are all 3 letters long).

* Reduce load time relocations in libstemmer by using a suitably sized char[]
  array in struct stemmer_encoding instead of const char*.

* Eliminate sentinel NULL entry at end of encodings array.  We know the length
  of the encodings list at compile time, so use that to limit iteration.

Testsuite
---------

* Expand runtime test coverage.  Partly addresses #301.

* Add runtime test coverage for `?` (debug command).

* Fix compilation errors in stemtest.c Introduced by v3.1.0.
    Currently stemtest doesn't get built or run by default, which is how this
    slipped in.
    Fixes #295, reported by Duncan Bellamy.
    We stopped compiling or running stemtest in v3.1.0.
    We can't actually run stemtest when runtime tests are enabled (it fails
    with "language `en' not available for stemming") but I meant to
    conditionalise it rather than just stop running it at all.
    Now `stemtest` is run on `make check` unless the tree has been switched
    to run runtime tests.  We now handle `compilertest` in the same way,
    since that is redundant to run when running runtime tests.
    Addresses the underlying reason why #295 wasn't detected right away,
    as a recurrence of #295 would now fail CI.

Documentation
-------------

* NEWS: Removed unused subheading in old entry.

Snowball 3.1.1 (2026-06-03)
===========================

Compiler changes
----------------

* Bug fixes:

  + Fix a segmentation fault after reporting an error for a string command
    not followed by a string variable name or string literal.  Bug introduced
    in 3.1.0.  Patch from Jerry James (#287).

* Compiler command-line options:

  + Emit an error for `-o -`/`-output -`.  Output to stdout is not supported
    because we need to generate multiple files for some target languages.
    We were interpreting `-` as a base filename to append extensions to, so
    we'd create `-.c` and `-.h` for C, but creating filenames that start with
    `-` seems unhelpful.

Generic code generation changes
-------------------------------

* Bug fixes:

  + Variable localisation was failing to check the expression on the RHS of an
    integer test for uses of a variable, so could incorrectly localise an
    integer variable whose value should have persisted between calls to a
    function.  This bug won't realistically manifest in real world Snowball
    code.

* Optimisations:

  + Inline some routines which are only used once.  This is done for routines
    consisting of a single non-compound command (or cases such as `not
    <boolean>` and `goto <grouping>` which we internally synthesise a
    non-compound command for).  Localisation of variables happens after
    inlining, so variables can now be localised in more cases.

  + `test next` and `not next` are both now simplified to a comparison between
    `cursor` and `limit` (like `not atlimit` and `atlimit`).  We already
    normalise `hop 1` to `next`, so `test hop 1` and `not hop 1` are also
    simplified in this way.

  + Simplify `not` applied to an integer test by removing the `not` and
    flipping the sense of the test (e.g. `not $(x > y)` becomes `$(x <= y)`)
    which results in simpler generated code.  More usefully in real world code,
    this also results in simpler generated code for `not atlimit` (since
    `atlimit` is converted `$(cursor >= limit)` or `$(cursor <= limit)`
    (depending on the current direction).

  + Canonicalise `delete` to `<-''`.  3.1.0 added a canonicalisation of `<-''`
    to `delete`, but if we instead make `<-''` the canonical representation
    then the generators can easily handle an empty string here specially (and
    if they don't then the results are still correct), we don't have to handle
    the delete token after tokenisation, and the optimisation of among when
    all actions are `<-` plus a literal string can also fire when some actions
    are `delete`.  (This case occurs in the stemmers for arabic, czech,
    estonian, hungarian, irish, italian, polish, porter and tamil.)

C++
---

* Code quality:

  + C++ runtime support is now fully hooked up and can be tested using
    `make check_cxx`.

  + Don't use `extern "C"` for functions which can throw because MSVC seems to
    treat such functions as if they were marked `noexcept`.

Javascript
----------

* Code quality:

  + Fix formatting of code generated by `among` of `<-` optimisation where we
    were emitting a literal `+` rather than increasing the margin size.  The
    `+` was actually harmless in practice, but the rest of the file would be
    misindented.

Python
------

* Other changes:

  + Skip classifier for Sesotho which isn't yet in the official list of
    trove classifiers.  Patch from Dmitry Shachnev (#289).

  + Add classifier to indicate support for Python 3.14.

Build system
------------

* The `generate` target added in 3.1.0 now overrides `FPC` and `MCS` to do
  nothing, to avoid compiling Pascal and C# code.

* Provide a way to run tests under valgrind:

    make RUN_STEMWORDS='valgrind ./stemwords' check

Testsuite
---------

* compilertest: Require syntax tests have stderr output - we are testing errors
  here, so we should really be checking the errors we get are the ones we
  expect.  We expect the compiler to fail, so testing stderr means we will
  catch unrelated failures.  This provides a regression test for #287.

Documentation
-------------

* NEWS: Fix formatting/wording of a few existing entries.

* README: List C++ as target language.

Snowball 3.1.0 (2026-05-22)
===========================

Compiler changes
----------------

* Bug fixes:

  + Fix segmentation fault if -syntax is used on a program with no code.

  + Fix segmentation fault on some assignment syntax errors.

  + Fix bug introduced in v3.0.0 with conversion of `among` starter.  If there
    were any commands after the among in the same command list then the among
    itself would get lost.  Not triggered by any current algorithms.

  + Clear name field when removing dead assignments.  This is visible in the
    syntax tree shown when command line option -syntax is used, but probably
    doesn't affect anything otherwise.

* Compiler command-line options:

  + Using `-` for the Snowball source file is now interpreted as stdin.

  + Improve comments generated by `-comments` to show more details of the
    corresponding Snowball code (e.g. variable names, arithmetic expressions,
    and literal strings).

  + Add `-coverage` option which enables a code coverage feature.  So far this
    tracks which among strings and functions are exercised, and which grouping
    characters are exercised. !

  + Support `-eprefix` for all target languages.  This is easy to do and
    provides a way to deal with externals which collide with keywords in the
    target language.  Our build system now uses `-eprefix _` for Python to make
    the `stem` external non-public (it is called by BaseStemmer method
    `stemWord()`) and we no longer hard-code prefixing Python externals with
    `_`.

  + Describe more options in `--help` output.

  + Sort target language options in `--help` output.

  + The `-o` option is now optional.  If not specified we now write output(s)
    to the same filename as the first source, but with a different extension
    (e.g. path/to/english.sbl -> path/to/english.c and path/to/english.h).

  + The `-o` option can now optionally include an extension so you can now
    write `-c++ -o path/to/foo.cxx` instead of `-c++ -o path/to/foo`, which can
    be more convenient (e.g. in `make` rules) and also provides an easy way to
    specify an alternative extension (for example, `.cxx`, `.cc` and `.cpp` are
    all extensions commonly used for C++ source code).

  + Reject `-vprefix` option for target languages which don't support it (it is
    currently only implemented for C/C++).

* Diagnostics:

  + Clean up and improve error reporting.

  + Improve line numbers reported for some errors and warnings by using the
    line number of an appropriate token rather than the current line number
    of the tokeniser (which is often the line after the command being warned
    about).

  + Improve recovery after various errors, trying to resynchronise based on
    what's more likely, and eliminating some additional irrelevant errors
    (including reporting the exact same error twice in some situations).

  + Emit warnings for uses of legacy Snowball language features.

  + The Snowball manual describes `integers (x)` as a declaration of `x` so we
    now warn:

      integer 'x' declared but not used

    rather than:

      integer 'x' defined but not used

  + 3.0.0 added a warning if the body of a `repeat` or `atleast` loop always
    signals `t` (meaning it will loop forever which is very undesirable for a
    stemming algorithm) or always signals `f` (meaning it will never loop,
    which seems unlikely to be what was intended).  This warning was added to
    the C generator, but has been moved to generic code so it is now issued
    regardless of the current target language.

  + Improve the wording of the warning if the body of a `repeat` or `atleast`
    loop always signals 't' to explicitly say this means the loop is infinite.

  + Improve warning message for unreachable code after `not`.

  + `$x = x + 1` cleared the initialised status of x (rather than just not
    setting it) which could lead to bogus warnings that `x` is never
    initialised.

  + The compiler no longer exits immediately after reporting a division by zero
    error in the Snowball code.

  + We now report a division by zero error for `$x /= 0` (this was meant to be
    already implemented but wasn't working due to a code typo).

  + More consistent wording of "is a no-op" warnings.

  + Warn that `insert ''` and `attach ''` are no-ops (and don't generate code
    for them).

  + Warn if a string used to define a grouping repeats characters.  There's no
    reason to do this, so it seems likely to be a typo.

  + Avoid sometimes reporting "-1 blocks unfreed".

* Optimisations:

  + Speed up processing larger Snowball programs by growing large string
    buffers exponentially to avoid a huge number of reallocations.  For
    example, this reduced the time to compile serbian.sbl to C by about 80%!

  + Optimise reading of input file when it is seekable (which it is in typical
    usage).  Non-seekable input files are still supported.

  + Optimise writing integers when generating code.  72% of integers we write
    are 0 to 9 and these are now written as a character.  Other values are now
    handled without a temporary buffer, avoiding a copy.  This reduced the time
    to compile serbian.sbl to C by about 8%, for example.

  + Optimise comparing among actions to find and merge equivalent actions.
    The comparison function used for this was carefully returning a full order,
    but actually we only need to know if the actions are equivalent or not
    which can be tested more efficiently.  For example, this reduced the time
    to compile serbian.sbl to C by about 2%.

  + We now precompute the possible signals from each command which means this
    is now done exactly once per command, whereas previously we could end up
    doing it many times for some commands in some cases.  The only functional
    change should we no longer make a pessimistic assumption if the function
    call depth reaches 100.  This is cleaner but is unlikely to make a
    difference for any real-world Snowball programs.

  + Handle possible_signals for string-$ which just passes on signals from its
    subcommand.  This doesn't affect code generation for any algorithms we
    currently ship.

  + We now only generate function bodies to a temporary buffer for target
    languages where we need to.  This makes the code a bit clearer and reduces
    the amount of copying of data so will make the Snowball compiler a little
    faster.  This change produces identical output for all current algorithms.

  + Tokenisation now decodes symbol tokens using switch statements.  We don't
    know the length of these tokens in advance, so the old approach of binary
    chop on a sorted list required searching the list multiple times with
    different possible lengths.  Alphabetical tokens are still decoded by
    binary chop.

* Code quality:

  + Remove unused routines and groupings from the program during the analysis
    phase, which avoids each generator having to have duplicate code to skip
    them.

  + Fix small memory leak if all uses of a name are eliminated.

  + Always use `snprintf()` instead of `sprintf()`.  If the buffer passed was
    too small we now emit an error rather than quietly using truncated output.

  + Fix GCC -Wcast-qual warnings in compiler and enable this warning by
    default.

  + Switch to using the standard C `bool` type in the code of the compiler.
    (The generated code still aims to require only C90.)

* Other changes:

  + Provide a simpler way to build a cut-down Snowball compiler.  The
    motivation here was to have a way to more quickly build a smaller Snowball
    compiler which only targets C.  Rather than have a DISABLE_xxx macro for
    each language, just check if TARGET_C_ONLY is defined, and only turn off
    the code to actually call the other generators which greatly reduces the
    amount of conditionalisation required.

Generic code generation changes
-------------------------------

* Bug fixes:

  + Fix code generated for `setlimit tomark` for all target languages to
    restore the limit correctly afterwards.  The bug was not triggered by any
    of the existing stemmers.

  + We no longer optimise repeat/atleast applied to goto/gopast on a (non)
    grouping.  This optimisation was flawed - it requires that the code in the
    loop preserves the cursor's value on failure, but the target language
    helper functions used here don't currently do that (they probably easily
    could so there's scope to reinstate this optimisation).

    Looking at the stemmers we ship, this affects the code generated for one
    loop in indonesian.sbl, but it happens the cursor value is overwritten
    immediately after this loop anyway.  The bug could affect non-shipped
    Snowball code though so isn't purely latent.  This bug was introduced in
    Snowball 3.0.0.

  + When generating target language literal strings we now always escape
    characters which can be problematic when viewing the generated source code.
    We always escape control characters U+007F to U+009F, non-breaking space
    U+00A0 (visually identical to a space), and U+0590 and above (as a crude
    way to avoid literal LTR characters in sources which can result in
    confusing rendering).

  + Fix line numbers given to various tokens (the line numbers previously given
    were at least of lines in the same command or the line after it).  These
    lone numbers can be seem in the target language comments generated when
    `-comments` is used.

  + Fix warning and simplification of code when `not` is applied to a command
    which always signals `t`.  Bug introduced in v3.0.0.  Fixes #271.

  + Warn and simplify `not` applied to a command which always signals `f`.

* Optimisations:

  + Add machinery to generate a Snowball variable as a local variable in the
    target language instead of it being "global" (typically a private class
    member in the target language).  This reduces the amount of state in
    stemmer objects, and typically reduces the overhead of accessing these
    variables a little.

    We now do this for integers and booleans in all target languages, and
    for strings in target languages where benchmarking seems to show it
    is faster (Dart, Go, JS, Pascal, PHP, Python).

    It's done for a Snowball variable which is only used in one routine, that
    routine doesn't (directly or indirectly) call itself, and the variable is
    set by any code path which leads to a use of the variable.  The mechanism
    which traces the code paths errs on being too conservative in some cases,
    but it's good enough for all instances in the code we ship, and is likely
    to handle the vast majority of real-world cases.  We issue an "info"
    diagnostic to report when a variable which is only used in one routine
    can't be localised - please report if you see this in real world code and
    we can try to enhance the code path tracing.

  + Tail-calling and similar optimisations can now work for non-trivial
    routines (previously they only worked for routines consisting of a
    single command and not enclosed in parentheses).

  + A grouping test at the end of a routine now generates simpler code.

  + A string test at the end of a routine now generates simpler code.

  + Optimise testing a boolean (optionally preceded by `not`) when used at the
    end of a routine.

  + Optimise an `among` with no commands at the end of a routine.

  + Generate simpler code for `not` applied to testing a boolean variable.

  + A `not` only needs to restore the cursor when its subcommand fails, so we
    now consider whether its subcommand can modify the cursor on failure
    (rather than whether it can modify the cursor at all).  Related to #226.

  + An `or` only needs to restore the cursor when a subcommand fails, so we
    now consider whether its subcommand can modify the cursor on failure
    (rather than whether it can modify the cursor at all).  We also now
    consider each subcommand individually, and only emit the cursor restore for
    those subcommands which need it.  (#226)

  + Both `and` and `or` only need to restore the cursor between sub-commands
    so no longer consider if the final subcommand might change the cursor.
    This makes some small improvements to the generated code for a few
    of the currently shipped algorithms.  (#226)

  + Handle more commands when checking if the cursor needs restoring - this
    improves the generated code for tamil.sbl a bit.

  + Single case amongs are now refactored to eliminate the `among` and so no
    longer call the among machinery.  Sometimes a single case among is the
    natural way to express a single rule in Snowball code as it can show
    commonality with rulesets with multiple rules, but it's inefficient to
    actually generate as an among.  Of the stemmers we currently ship, this
    improves code generation for arabic, estonian, greek and lithuanian.

  + Avoid unnecessary cursor update in among helpers.  We only need to update
    the cursor on success, but were unconditionally doing so after calling an
    among function.

  + Handle more commands in repeat_score().  None of these help code generation
    for any currently shipped algorithms, but they are valid to optimise here.

  + Canonicalise `<-''` to `delete`.

  + Simplify some cases of compound assignment operators when the argument is
    (or can be simplified to) a constant integer, like we already do for
    arithmetic expressions.  For example, `$x += len '{a"}' - len 'a'` is a
    no-op when using a fixed-width encoding.

  + Canonicalise `fail C` to `false` in some cases where `C` has no
    side-effects.

  + Removing unreachable code could leave single-entry `and`/`or` nodes
    which could result in generating target language code with unused
    variables.  These nodes are now replaced with their subnode.

  + Eliminate `true` below `and`/`false` below `or`.  These are unlikely to
    appear verbatim in real programs, but can be created by optimisations, and
    also can appear in runtime tests, leading to the generated target language
    code having unused variables and/or unreachable code.

  + Canonicalise setmark, atmark and atlimit by converting `setmark x` to
    `$x = cursor`, `atmark x` to `$(cursor == x)` and `atlimit` to either
    `$(cursor >= limit)` or `$(cursor <= limit)` (depending on whether we're
    in backwardmode or not).  This means the target language generators have
    three fewer commands to handle, and also gives us tail-calling of `atlimit`
    and `atmark x` (there's a tail-callable use of `atlimit` in the Turkish
    stemmer).

  + Remerge among actions after optimisation.  It seems hard to fully move the
    code to merge them later, but we can check for actions which have become
    equivalent to `true` or to other actions after optimisation but before we
    generate code.

* Code quality:

  + Find Snowball routines which are not reachable by calling an external.  We
    no longer generate code for such routines, nor for variables and groupings
    which are only used in them, which helps to avoid "unused" warnings in the
    generated target language code.

  + If the sub-command of `repeat`/`atleast` always signals `t` or always
    signals `f` we now prune the rest of the current command list, and simplify
    the command for always `f` (c_repeat -> c_do; c_atleast -> c_bra).  These
    changes help to avoid generating redundant target language code which can
    trigger errors or warnings.

* Other changes:

  + `delete` and `<-` now update the slice end (see the "Snowball Language
    Changes" section).

Ada
---

* Bug fixes:

  + Ada variable names are case-insensitive, so if two Snowball names of the
    same type differed only by case we would generate Ada code with a name
    collision.  We now avoid such collisions by adding a counter after the type
    code for the second and subsequent names that differ only by case.

  + Ada stemmer names are now prefixed with `S_` so `or.sbl` now generates
    stemmer `S_or`, avoiding a name clash with an Ada keyword.

  + Fix Ada code generated `for setlimit tomark p`.  This affected the
    generated code for the Lithuanian stemmer, but it appears by luck in this
    case the bug didn't actually affect the stemmer's output for any input.

  + Fix `setlimit` ... `repeat` bug.  The generated Ada code was running the
    code to recover from a failure inside a `repeat` loop twice due to a
    missing line of code compared to other generators.  In `backwardmode`, the
    failure code happens to be idempotent so running it twice doesn't cause a
    problem, but in forwards mode this results in the cursor getting double
    adjusted if the length of the stem has changed due to insertions, deletions
    or substitutions.  None of the existing algorithms use `setlimit` in
    forwards mode, so they're unaffected by this bug.  Fixes #275.

  + Fix overcopying in string replacement code.  The code to move the tail
    up/down was copying one byte too many.  We're working in a 1024 byte
    fixed-length string buffer, and the maximum allowed input word is one byte
    shorter, so it seems this was harmless in practice.

  + Allow characters <32 and 127 in string literals.

  + `=S` can no longer result in the slice ends becoming negative and
    triggering a CONSTRAINT_ERROR (the slice is now specified to be unset
    after `=S` - see the "Snowball Language Changes" section).

  + Fix Ada code generated for string-$ which was actually partly Pascal
    code (the Ada generator was originally based on the Pascal one) and
    didn't even compile.  To fix this, Snowball string variables in Ada the
    same way as the current string.  This means they now take up more space (a
    fixed 1KB), but a typical Snowball program has either no string variables
    or just one so the overhead seems acceptable.

  + Fix matching of an empty string variable.  This valid Snowball code would
    trigger "failed precondition" in Ada:

      externals (stem)
      strings (s)
      define stem as ([] ->s s)

  + Fix assumption that there's a single external called "stem".

  + Fix incorrect assumption that an among containing the empty string
    always matched, even if the empty string had a gating function.
    This construct is not used by any existing stemmers.

* Optimisations:

  + Avoid calling among helper when the among contains only strings which are
    one byte long, no among functions are used, and there are no actions.

* Code quality:

  + Fix indentation of generated grouping tables.

  + Rename Context to Z in runtime code.  This now matches variable naming in
    the generated Ada code (and also the C runtime and generated C code).

  + Eliminate redundant limit check (the Skip_Utf8 helper also checks the
    limit).  Looking at the history this check is a left-over from when the
    generated code directly incremented the cursor.

  + Emit Ada literal strings without redundant empty strings between adjacent
    escaped bytes.

  + Generate dummy loop around `or`, which allows us to handle a sub-command
    succeeding with Ada `exit` rather than `goto`, which seems clearer.

  + Avoid creating unused labels.  This is just a cosmetic improvements - there
    are no longer mysterious gaps in the numbering of labels in the generated
    code.

  + Avoid generating unreachable `exit`.

* Other changes:

  + Implement support for `?` (debug command).  The code we generate for this
    case is gnat-specific, but previously the code generated didn't compile so
    working with one implementation seems a step forwards.  The `?` command can
    now be used to debug Ada, and someone with actual Ada knowledge can now
    more easily step in and provide a portable replacement.

C/C++
-----

* Bug fixes:

  + Maintain invariant that the C variable corresponding to a Snowball string
    variable is non-NULL.  Previously we would release and NULL out the entry
    in some error cases, but elsewhere the code was assuming the value was
    non-NULL.

  + Fix invalid code generated for `setlimit`.  This doesn't happen for
    `setlimit tomark` (which is the only way `setlimit` is used in the stemmers
    we currently ship.  Bug introduced in v3.0.0.

  + Fix codegen for `hop` with constant argument.  We were relying on the
    cursor being restored on failure by the code which handled that failure,
    but if that code is a repeat or atleast command that it has an optimisation
    which assumes `hop` won't do this.  This means we generated incorrect C
    code for some cases where `hop` was used inside `repeat` or `atleast`.
    This doesn't functionally affect any of the stemmers we currently ship.
    Bug introduced in v2.1.0.

  + Fix bug in code generated when `-vprefix` is specified, introduced in
    Snowball 2.1.0.

  + Fix incorrect assumption that an among containing the empty string
    always matched, even if the empty string had a gating function.
    This construct is not used by any existing stemmers.

* Optimisations:

  + Rework how non-localised variables are stored, which eliminates an
    indirection on every access to such a variable, and also avoids some extra
    allocations (one if a stemmer has any non-localised integer or boolean
    variables, and another if the stemmer has any string variables).  So it
    uses a bit less memory, it makes creating and destroying a stemmer faster,
    and it also makes stemming a bit faster (though only by ~0.1% for the
    English stemmer on our sample vocabulary).  The `-vprefix` option now
    generates getter functions rather than using macro magic, which means the
    syntax for accessing Snowball variables from C has changed.

  + We now maintain the invariant that SN_env's p member is non-NULL, which
    simplifies the runtime code.

  + We now have a specialised implementation of the slice_del() runtime helper.
    Deleting the slice is a fairly common operation, and can be done more
    simply than via a generic replace_s() with an empty replacement string.
    This speeds up the English stemmer by about 1% on our test vocabulary.

  + Avoid calling among helper when the among contains only strings which are
    one byte long, no among functions are used, and there are no actions.

  + Only fetch SIZE() in replace_s() if we need it.

  + Don't return adjustment from replace_s() runtime helper since calculating
    the adjustment in the one caller where we actually want it is just one
    integer addition and one integer subtraction, and that turns out to be
    slightly more efficient as well as simpler.

  + Move check for negative hop from runtime to generated code.  This means we
    can omit it for hop with a constant argument, which is all uses of hop in
    the stemmers we currently ship.

* Code quality:

  + The generated header is now included from the generated C/C++ source file
    (which seems cleaner than the previous approach of generating the same
    prototypes in the header and source file).

  + The implementation of among functions has changed.  Previously we stored
    a function pointer in struct among, but that requires relocation when the
    code is in a dynamic library, which adds load-time overhead and means
    the among structures can't be put in a read-only section.

    We now store an integer index instead, and pass in a pointer to
    a dispatcher function when calling the find_among()/find_among_b() helper
    which gets called when this index is non-zero.  The value of the index is
    stored in z->af so the dispatcher function can use it.

    If only one unique function is used in an among, we can just pass this to
    find_among() as the dispatcher which reduces the overhead for this common
    case.

    Profiling with cachegrind suggests this change adds a small overhead
    to algorithms which use among functions - currently finnish and hindi
    (and also lovins, but that's really only of academic interest and is not
    enabled by default).

  + Avoid long string in C source.  C90 only guarantees support for literal
    strings up to 509 characters.  Fixes GCC -Woverlength-strings warning.

  + Avoid C23 feature in C runtime code, introduced in Snowball 3.0.0.
    Initialising with empty braces was only standardised in C23 (though
    seems to be widely supported as an extension).

  + Fix code generated for `setlimit` to be C90.  Bug introduced in v3.0.0, but
    isn't triggered by any of the stemmers we currently ship.

  + Fix -Wshadow warning for nested string-$ use.  We were generating code
    using a C variable with the fixed name `failure` - now an integer suffix is
    appended, and we only emit the variable in cases where the subcommand
    signal isn't known at compile time.

  + Generate `do {`...`} while (0)` around `or` code, which allows us to handle
    a sub-command succeeding with `break;` rather than having to use `goto`,
    which reduces the number of labels used and makes the generated code a bit
    easier to follow.

  + C comments are now generated for `(` and `do` when `-comments` is used.

  + We now generate `+=` or `-=` for `hop <constant>` (instead of something
    like `z->c = z->c + 2`).  The C compiler should treat both the same, but it
    arguably makes the generated code a little clearer.

* Other changes:

  + C++: The `-c++` option used to generate exactly the same code as for C,
    except with extension `.cc` instead of `.c` but now:

    - C++ classes are generated.
    - C++ `bool` is used for Snowball booleans.
    - Loop variables are declared inside `for (`...`)`.
    - Allocation failures and internal errors (e.g. slice_check() failing)
      throw a C++ exception - this is a bit simpler and more efficient that the
      C code approach of returning -1 which then has to be checked for and
      propagated through the generated code.

  + Snowball's debug command (`?`) now works out of the box (previously you
    have to adjust a `#if 0` preprocessor conditional in the runtime code).

  + Rename `runtime/header.h` (which really seems too generic, and is also easy
    to confuse with `compiler/header.h`) to `runtime/snowball_runtime.h`.  We
    expect most users will be using the C stemmers through libstemmer and so
    won't be affected by this.

C#
--

* Bug fixes:

  + Fix code generated for `<-s`.  This is not used by any of the stemmers we
    currently ship.  Test case based on one from ajroetker in #270.

  + Fix code generated for string-$.  This feature is not used by any of the
    stemmers we currently ship.

  + Fix assumption that there's a single external called "stem".

* Optimisations:

  + Use Debug.Assert() in slice_check() runtime helper.  Previously the runtime
    code wrote a diagnostic message and continued if one of these checks
    failed, but failures should only happen with a Snowball program containing
    logic errors, or for bugs in the Snowball compiler or its runtime (or
    possibly in the C# compiler, runtime, OS, hardware, etc).  Therefore an
    assertion seems an appropriate choice, and means the check is not enabled
    for a production build, which seems more helpful overall.  See #242.

* Code quality:

  + Eliminate duplicates from groupings.  We currently implement these for C#
    with a linear string search, and a side-effect of this change is that the
    grouping string is now sorted, which will affect the time taken to look
    up different characters in an arbitrary way (none of the Snowball sources
    seem to try to list characters in frequency order).  Really C# should be
    fixed to use an O(1) lookup like other target languages.

  + The implementation of among functions has changed.  We now store an integer
    index in the Among class, and pass a dispatcher function to the among
    helper method.

    If only one unique function is used in an among, we can just pass this to
    the helper method as the dispatcher which reduces the overhead for this
    common case.

    Crude profiling with `time make check_csharp` suggests this doesn't harm
    performance (perhaps a little faster, but maybe just within the noise).

    The main benefit is all Among arrays can now be static, which previously
    we wasn't possible for those which used among functions (#146).

  + Remove unused return value from Stemmer.Replace() runtime helper.

  + Fix inaccurate doc comments on runtime functions.

* Other changes:

  + csharp_stemwords: Speed up output to stdout.

  + csharp_stemwords: Don't write the chosen stemmer to stdout.  This is not
    really useful information, and breaks sending the stemmed words to stdout
    because they're preceded by extra output.

  + csharp_stemwords: Try to open input before output so we don't leave an
    empty output file behind if we can't open the input file.

Go
--

* Bug fixes:

  + Fix code generated for non-constant hop.  A non-constant integer expression
    has type `int` in the generated Go code, but the hop helpers expected
    `int32`.  For a constant hop this worked because Go integer literals are
    untyped, so will convert to `int32`.  To fix this, the helpers now take
    `int` instead of `int32`.

  + Fix code generated if `minint` or `maxint` is used.  In this case we were
    generating `use std::usize;` near the start of the Go code, but that's
    actually Rust code and a hangover from the Go backend being originally
    based on the Rust one.

  + The Go code generated for `->` was incorrectly signalling `f` if the
    slice was empty.  Luckily this case is not exercised by any current
    algorithms.  See #242.

  + Fix code generated for string-$ (which isn't used by any of the algorithms
    we currently ship).

  + A snowball `external` could not previously be called from within the
    Snowball program.  This is allowed by the Snowball language, but none of
    the shipped stemmers do this, and it's unlikely any stemmer would.  Perhaps
    it's useful if you use Snowball for other string-processing tasks.

  + Fix handling of `minint` and `maxint` - we were generating some code copied
    verbatim from the Rust generator for this case which was not valid Go.
    (These are not used by any of the algorithms we currently ship.)

* Optimisations:

  + Reuse `env` in stemwords which is measurably faster than creating a new one
    for every word.

* Code quality:

  + Eliminate unnecessary semicolons from generated code.

  + Fix formatting of generated code.  The code gets run through gofmt which
    was fixing up these issues, but better to generate the code cleanly to
    start with.  The only things which gofmt now changes are that it indents
    variable names to align in adjacent variable declarations, and a couple of
    things which are apparently for compatibility with older versions of Go.

  + Runtime helpers SliceDel() and SliceFrom() always returned true, but the
    generated code included failure checks in case false was returned.
    These helpers no longer return anything, and the checks are gone.

* Documentation:

  + Recommend that users reuse an `env` since this is measurably faster than
    creating a new one for every word.

* Other changes:

  + Remove `-gopackage` option from compiler.  Use `-package`/`-P` instead
    (`-gopackage` has just been an alias for these since Snowball 2.0.0).

Java
----

* Bug fixes:

  + Generate correct Java code for ASCII control chars in string literals.

  + Fix code generated for string-$.  As part of this fix, we now use char[]
    for string variables as well as the current string, which makes it much
    simpler to switch to working on a string variable and back.  Fixes #252.

  + Fix assumption that there's a single external called "stem".

* Other changes:

  + The generated Java classes no longer implement Serializable.  This support
    was added in 2016, but in 2026 this approach to serialization in Java is
    apparently no longer used due to security problems.  Fixes #255.

Javascript
----------

* Bug fixes:

  + Fix `->` to work when the slice is empty - previously it incorrectly
    signalled `f` for this case.  Luckily this case is not exercised by any
    current algorithms (#242)

  + Generate public functions for all externals.  Patch from simlrh (#258).

  + Fix code generated for string-$

* Optimisations:

  + Use startsWith()/endsWith() in eq_s()/eq_s_b().  This is quite a bit faster
    as it avoids slice() creating a temporary string (e.g. measured a reduction
    of ~17% wallclock time for tamil on the test vocabulary, taking the fastest
    of 5 runs before and after).

  + Optimise among when all actions are `<-` with a literal string.  We now
    generate a single call to slice_from() with the argument obtained by
    indexing into an array of literal strings.  This is perhaps faster, albeit
    not by much, but it definitely results in smaller code, which is helpful
    for in browser use.  See #227.

  + The substring_i member in the Among class is now an offset from the current
    index, and now zero in the common case where there's not another string
    which is a sub-prefix/sub-suffix.  We've also swapped the order of elements
    so we can omit this in the common case when it is zero and there's no among
    function).  This reduces the size of the generated Javascript code (even
    after minification).  Fixes #236.

  + Change slice_check() to assert its conditions.  In C we must not perform
    string slicing if slice_check() fails because that could result in writing
    outside of the allocated buffer, but it's not problematic in this way for
    Javascript, and the situations which slice_check() checks for should only
    happen with a Snowball program containing logic errors, or for bugs in the
    Snowball compiler or its runtime (or possibly in the Javascript
    interpreter, OS, hardware, etc).  Therefore assert() seems an appropriate
    choice.

* Code quality:

  + Convert to using Javascript modules and classes.  The way among functions
    are called has been reworked to allow this, copying the approach now used
    for C and C# (#234, #240).  Patches from Adam Turner and Titus Ng.

  + Adjust generated code to work with deno, and suppress a few deno warnings
    which are hard to avoid in generated code.

  + Avoid generating blocks around failure handling.  The failure handle code
    is always a single statement (and if we ever needed more than a statement
    for some situation then we could arrange to add a block for just those
    situations).  This significantly reduces the size of the generated JS code.

  + Always inline code for `=>`.  The code is not much longer than the
    call to a helper function in BaseStemmer.  Also in 3.0.0 we deprecated `=>`
    and nothing we ship contains this command, so removing it from BaseStemmer
    reduces the total code size a little.

  + Rename BaseStemmer's internal `cursor` property to `c`.  Unfortunately,
    `cursor` is a DOM property, so Javascript minifiers are cautious about
    renaming it to avoid breaking code.  The name `c` matches the naming we use
    for C, Ada and Pascal.

  + Generate smaller code for hop by constant.  All current uses of hop in the
    stemmers we ship have a constant argument, so avoid using a temporary
    variable in these cases.

  + Optimise `+=1` to `++`, `-=1` to `--`.  These are a byte shorter, and it
    seems Javascript minifiers don't do this for us because it's not a safe
    transformation unless the minifier can deduce that the variable can't hold
    a string.

  + Improve temporary var naming and use.  These variables don't need unique
    generated names now we're declaring them as `const` which has more sensible
    scoping rules than `var`.

  + Generate smaller code for `insert` and string-`=`.  In some cases we know
    we have the value of member variable `this.cursor` in local `const c` so
    use the latter instead.

  + Use triple equality for Javascript.  Patch from Adam Turner.

  + Fix position of grouping type comment which is now placed consistently with
    other type comments.

  + Use `a` instead of `among_var` in generated code.  This reduces the size of
    the generated code, which is helpful if a minification step isn't being
    used.

  + Consistently cuddle braces in runtime code.  The style wasn't entirely
    consistent before, and cuddling braces matches the generated Javascript
    code and the Snowball C code.

  + Generate block around case to bound the scope of `const` and `let` within
    the case.

  + Use `let` in README example.

  + Use `let` consistently in stemwords.js.

  + Initialise integer Snowball variables - we annotate them as being type
    "number" so we shouldn't let them have value undefined.  Patch from Adam
    Turner.

  + Improve/fix typescript annotations in runtime and generated code.

  + Annotate runtime with @ts-expect-error.  It doesn't seem to be possible to
    express the types fully in some places, but the invariants we require are
    ensured by the Snowball compiler.  Annotating the expected errors allows
    unexpected type checking errors to be be more easily seen, and they are
    now fatal in CI.

  + Use `===` and `!==` in stemwords.js.  Patch from Adam Turner.

* Other changes:

  + Make stemmer subclasses anonymous and export them by default.  This makes
    creating a stemmer object easier as you only need to build the filename of
    the stemmer subclass, and not also its class name.

  + Adjust interpretation of `-parentclassname` option.  We supply the JS
    snowball runtime so being able to specify a different base class name
    doesn't seem very useful, so instead interpret this as the name to import
    the base class as in generated stemmers.  It now defaults to just `B` which
    reduces the size of the generated stemmer code a little (even after running
    it through most Javascript minification tools).

  + Improve stemwords.js option parsing.  Make `-i` and `-o` optional to match
    other target language versions of stemwords.  Eliminate the check that
    there are at least 3 command line arguments as we don't require any now.
    If we encounter an argument we don't understand, we now report it and show
    the usage message (previously we silently ignored it).  We now exit with
    status 1 if there's a problem parsing the command line.

  + stemwords.js: Emit help message in one console.log.  Patch from Titus Ng
    (#221).

Pascal
------

* Bug fixes:

  + We were generating invalid Pascal code when tail-calling or calling a
    routine which always fails.  Neither case is currently exercised by any
    stemmers we ship and generate Pascal code for (the Pascal generator
    currently only supports iso-8859-1).

  + Fix code generated for string-$ (which isn't used by any of the algorithms
    we currently ship).

  + Fix assumption that there's a single external called "stem".

* Code quality:

  + Merge EqS and EqV runtime functions.  We can get the length of a Pascal
    AnsiString `s` cheaply with `Length(s)` so there isn't a need to pass in
    the length in the string literal case.

  + Eliminate `While` in code generated for `repeat`/`atleast`.  Pascal lacks
    `Continue` (at least as a standard feature) and this loop only exists so we
    can jump back to its start with `continue` in other languages - we have a
    `Break;` at its end so it doesn't loop in the normal way.  In Pascal we
    generate a label before the loop and use `goto` to continue iterating, so
    we can get rid of the Pascal loop entirely.

  + Use `Break` instead of `Goto` in code generated for `go`/`gopast`.

  + Generate dummy loop around `or` so we can handle a sub-command succeeding
    with Pascal `Break` rather than `Goto`, which seems clearer.

  + Avoid generating `Repeat` ... `Until True` dummy loops which are not
    actually needed.

  + Fix problem introduced in v3.0.0 with formatting of code generated for
    `go`/`gopast` applied to a grouping.

  + Switch to a simpler name mangling system.  Pascal variable names are
    case-insensitive but Snowball names are case-sensitive.  We used to address
    this by encoding the case of letters into a prefix on the name but that can
    generate long and ugly names in some cases (e.g. integer Foo_Bar ->
    IUllU_Foo_Bar).  We now avoid collisions by adding a counter after the
    type code for the second and subsequent names that differ only by case
    (so Foo_Bar is only mangled if there's another integer which differs
    only by case which is declared before it, and even then just becomes
    something like I2_Foo_Bar).

  + Emit Pascal literal strings without redundant empty strings between
    adjacent escaped bytes.

  + The -comments option now includes the values of string literals, so has
    been changed to generate "rest of line" comments (starting `//`) rather
    than block comments (delimited by `{` ... `}`) so that string literals
    containing `}` don't need escaping.  We were already using `//` comments in
    the Pascal runtime so this shouldn't harm portability.

Python
------

* Bug fixes:

  + Fix `algorithms()` when forwarding to PyStemmer.  It looks like this has
    never worked as the code has been like this since it was merged, and we
    were forwarding to a method which PyStemmer doesn't provide and never seems
    to have provided.

  + stemwords.py: Make -i and -o optional.  The command syntax already
    suggested they were, but actually we gave an error if they were omitted.

  + Fix code generated for string-$ (which isn't used by any of the algorithms
    we currently ship).

  + Fix `->` to work when the slice is empty - previously it incorrectly
    signalled `f` for this case.  Luckily this case is not exercised by any
    current algorithms (#242)

  + Remove deprecated licence classifier which now triggers a deprecation
    warning from Python's setuptools.  We already specify the licensing in the
    now preferred way via `license=` with a SPDX licence expression.

* Optimisations:

  + Optimise single-character string literal checks in the same way we already
    do for C.  This seems to be measurably faster (tested with Turkish which
    has lots of single character literal tests).

  + Groupings are now implemented via a Python set, or a string for small
    groupings.

  + Eliminate use of exception in code generated for `or`.  We can instead wrap
    the code in a loop and use `break`.

  + Eliminate use of exception in `goto` and `gopast`.  We can just use `break`
    here to exit the `while` loop we're also inside and move the `except` from
    the previous `try` onto the `while`.

  + Avoid using a temporary for `hop` with a constant argument as benchmarking
    with timeit shows this is faster.

  + Optimise string test by using startswith()/endswith() with suitable
    start/end parameters which avoids creating a temporary substring and avoids
    an explicit limit check.  This speeds up artificial testcases consisting of
    `goto 'the'` by 10%.

  + Optimise among when all actions are `<-` with a literal string.  We now
    generate a single call to slice_from() with the argument obtained by
    indexing into an array of literal strings.  See #227.

  + Reduce overhead of code to forward to PyStemmer, both when forwarding and
    when using the pure Python stemmers.

  + Reuse exception classes much more.  This reduces the number of labN classes
    we need by 142 over all the current stemmers.

  + Change slice_check() to assert its conditions.  In C we must not perform
    string slicing if slice_check() fails because that could result in writing
    outside of the allocated buffer, but it's not problematic in this way for
    Python, and the situations which slice_check() checks for should only
    happen with a Snowball program containing logic errors, or for bugs in the
    Snowball compiler or its runtime (or possibly in the Python interpreter,
    OS, hardware, etc).  Therefore assert() seems an appropriate choice.

* Code quality:

  + Use _ as dummy loop variable.  We don't use the loop variable's value, and
    the loop itself tracks the current iteration so generating nested loops
    using `_` as the loop variable works correctly.

  + Avoid mysterious gaps in the numbering of variables in the generated code.
    This was already done for the other languages, but I missed Python it
    seems.

  + Avoid generating unused lab0 class for a Snowball program which doesn't use
    any failure labels.

  + Avoid generating a blank line at start of the body of a Snowball `loop`.

  + stemwords.py: Replace deprecated `codecs.open()` with built-in `open()`.
    Patch from Dmitry Shachnev.

* Documentation:

  + Remove unnecessary semicolons from Python code in docs.

* Other changes:

  + Remove Python 2 support.  We stopped officially supporting it in Snowball
    2.1.0, but now we've actually stripped out support.  Versions of Python ≥
    3.3 continue to be supported.  Patch from Dmitry Shachnev (#212).

Rust
----

* Bug fixes:

  + Fix code generated for string-$ (which isn't used by any of the algorithms
    we currently ship).

  + A snowball `external` could not previously be called from within the
    Snowball program.  This is allowed by the Snowball language, but none of
    the shipped stemmers do this, and it's unlikely any stemmer would, but
    perhaps it's useful if you use Snowball for other string-processing tasks.

  + The generated code previously treated an empty string returned by
    slice_to() as an error, but this was buggy since if the slice is empty
    the return value will be an empty string.  The helper doesn't try
    to signal an error with an empty string so we can just drop this
    check.  Luckily this case is not exercised by any current algorithms.
    See #242.

  + Fix incorrect assumption that an among containing the empty string
    always matched, even if the empty string had a gating function.
    This construct is not used by any existing stemmers.

* Optimisations:

  + Avoid calling among helper when the among contains only strings which are
    one byte long, no among functions are used, and there are no actions.

* Code quality:

  + Fix formatting of code generated for `goto`/`gopast` applied to a grouping
    or inverted grouping.  This is just a cosmetic problem - functionally it
    was correct.  The poor formatting was introduced in v3.0.0.

  + Runtime helpers slice_del() and slice_from() always returned true, but
    the generated code included failure checks in case false was returned.
    These helpers no longer return anything, and the checks are gone.

  + Generate space after condition in integer test (purely cosmetic).

New Code Generators
-------------------

* Add Dart generator from Ryan Heise (#156, #250).

* Add PHP generator from Tim Whitlock and Olly Betts (#243).  Requires PHP 8.3
  or later, which allows us to use typed class constants.

* Add Zig backend from AJ Roetker.  Requires Zig 0.16.0 or later.

Snowball Language Changes
-------------------------

* `delete` and `<-` now update the slice end.  The manual said that after
  `[` and `]` "the slice ends will retain the same values until altered",
  which doesn't make it clear what happens for operations which modify the
  text the slice ends are in.

  The existing handling here was inconsistent between commands: `delete`
  and `<-` left the slice ends on the same numeric positions, while
  `attach` and `insert` adjusted the slice ends to leave the slice marking
  the equivalent substring of the updated string.  When working in UTF-8
  the slice end could end up in the middle of a multi-byte character after
  `delete` or `<-`, which seems especially undesirable.

  I talked this over with Martin Porter and we've agreed that it makes
  sense for `delete` and `<-` to also update the slice ends (in fact only
  the right end needs adjusting) and I've clarified the wording in the
  manual.

  Existing algorithms we ship don't rely on what the slice is set to after
  these commands.

* The slice is now specified to be unset after `=S` (so the same state as at
  the start of the program).  Previously Snowball attempted to adjust the slice
  after `=S`, but there isn't an obvious adjustment in general because it can
  replace part of the content of the slice.  Martin said he'd not thought of
  this case, and we've concluded it's best to adjust the Snowball language
  definition.

New stemming algorithms
-----------------------

* Add Czech stemmer from Olly Betts and Jim O’Regan (#151).

* Add Persian (Farsi) stemmer from Saeid Darvish (#181).

* Add Polish stemmer from Dmitry Shachnev (#245).

* Add Sesotho stemmer from Kamohelo Lebjane (#260).

Behavioural changes to existing algorithms
------------------------------------------

* Danish:

  + Adjust to handle apostrophe (#187).

  + Restrict undoubling to valid cases.  Coverage showed that a number of the
    consonants we would undouble never occur in our Danish vocabulary.  Testing
    a larger list didn't find any matches for Danish words either, so restrict
    the undoubling which reduces the potential for damage to foreign words and
    should be a little more efficient.

* English:

  + Restore exception for `skis` so it stems to `ski`.  This reverts a
    change made erroneously in Snowball 3.0.0.

  + Improve the stemming of some words starting `inter`:
    - We now avoid conflating intern, internal, international and
      internment.
    - We now conflate interfere/interferes/interference with
      interfered/interfering.
    - The stem of `interval` is now `interval` rather than `interv`, which
      is mostly a cosmetic change as no unrelated words stem to `interv`.

* Estonian:

  + Handle apostrophe (#187).

* Finnish:

  + Handle apostrophe (#187).

  + Improve fallback from illative rules.  If a word ends -han, -hen, -hin,
    -hon, -hän or -hön but the vowel before does not match we were not removing
    a suffix in case_ending, we now fallback to handling as a genitive and
    remove -n.  This changes how we handle about 90 words - almost all for the
    better, most of the rest seem neutral changes.

  + Allow "ø" to match with -hön as this is seen with Norwegian place names,
    e.g. Bodøhön.

  + Remove illative form -hun.  This improves the stemming of 14 words in
    our test vocabulary.

* German:

  + Handle apostrophe (#187).

* Italian:

  + Handle elisions (#187).

* Lithuanian:

  + Don't remove -er- before normal suffixes.  These aren't real grammatic
    suffixes and seem to have been included mainly to try to conflate ancient
    forms of the Lithuanian word for "sister" (e.g. "sesers") with modern forms
    (e.g. "sesė").  We weren't even doing a complete job there however as
    "seserimis" and "seseris" were not handled.  Removing these suffixes
    entirely means we no longer try to conflate the ancient and modern forms
    here, but at least all the forms of the old word get grouped, as do all
    forms of the new word.  The stemming for ~150 other words is also improved,
    without obvious downsides.  Patch from Justas Sakalauskas (#263).

  + Remove trailing apostrophe as final step - an apostrophe is sometimes used
    to separate a Lithuanian ending on an international word (#187).

* Norwegian:

  + Adjust to handle apostrophe (#187).

* Polish:

  + Remove optional apostrophe after removing suffix.  Polish uses an
    apostrophe to separate loanwords from native suffixes.  (The correct use is
    to mark the elision of the final sound of a loanword before a Polish
    inflectional endings, but it's also often used with any loanword) (#187).

Optimisations to existing algorithms
------------------------------------

* English:

  + Optimise -eed, -eedly handling by performing the much cheaper R1 check
    before the among of exceptional cases.

* Esperanto:

  + Eliminate use of among functions.  It's easy to avoid them, and they come
    with a performance overhead in some target languages.  For C, the new
    version is 0.09% faster (from cachegrind estimated cycle count).

* Indonesian:

  + Avoid use of among functions, which gives a 1.9% speed up for C (from
    cachegrind estimated cycle count).

* Lithuanian:

  + Minor simplification/optimisation by relying on Snowball restoring the
    cursor on failure.

* Turkish:

  + Simplify `not test C` to just `not C`.  If C succeeds, then the `not` fails
    and the cursor will get restored by whatever handles that signal.

Code clarity improvements to existing algorithms
------------------------------------------------

* Finnish: Rename `V1` and `LONG` to match the names used in the algorithm
  description on the website.

* Italian: Eliminate use of legacy among starter.

Build system
------------

* The default flags used with `ar` are now `-cr` instead of `-cru`.  Many
  Linux distros configure `ar` to use option `D` (deterministic mode) by
  default, which was triggering a warning that option `u` is ignored.
  Option `u` is just a minor optimisation for the case where the archive
  already exist and only some object files have change, so it seems best
  to just not try to use it and avoid the warning.

  Make variable `ARFLAGS` can now be used to specify flags to use with `ar`,
  so if you want to continue using `-cru`, you can use:

    make ARFLAGS=-cru

  If `D` is on by default in your `ar`, you'll actually want:

    make ARFLAGS=-cruU

* Add comment documenting how to use iconv.py (simple pure-Python alternative
  which allows running the testsuite without iconv installed).

* `make clean` now removes all built files for all target languages, and is
  now tested by CI to ensure this doesn't regress.

* Make "make check_utf8" parallel-safe by avoiding writing the stemmed output
  to disk by default (except for Arabic).  To get the output saved as tmp.txt
  on error for debugging you can now use: `make SAVETMP=1 check_utf8`.  Patch
  from Adam Turner (#237, #238).

* Ada: Fix parallel build by adding missing dependency from .adb to the
  corresponding .sbl file (#237, #238).

* Go: Use `$(go)` for `go generate` as well.

* Python: Omit output "(THIN_FACTOR=)" if set empty.

* Add SNOWBALL_FLAGS, intended to allow passing options such as `-comments`
  and `-coverage` during development and debugging.

* Add make targets to assist comparing generated code before and after a
  compiler change: `baseline-create`, `generate` and `baseline-diff`.

* We now have CI testing that the Snowball compiler builds as C99 (we were
  already testing that the generated C code builds as C90).  Fixes #283,
  reported by Domingo Alvarez Duarte.

Testsuite
---------

* New testsuite for the Snowball compiler which tests parsing, errors and
  warnings.

* New runtime testsuite which tests the implementation of Snowball language
  features in each supported target language.  These provide something much
  more like a proper set of unit tests rather than relying on checking all the
  algorithms produce the expected output to validate all the target language
  generators.  These tests are run with -comments on to provide some test
  coverage for this option.  Fixes #157.

* stemtest: Add more number testcases, relocated to here from finnish/voc.txt.
  They're better by stemtest as we want to avoid any stemmer damaging numbers,
  and testcases here can easily be run for all stemmers.

Snowball 3.0.1 (2025-05-09)
===========================

Python
------

* The __init__.py in 3.0.0 was incorrectly generated due to a missing
  build dependency and the list of algorithms was empty.  First reported by
  laymonage.  Thanks to Dmitry Shachnev, Henry Schreiner and Adam Turner for
  diagnosing and fixing.  (#229, #230, #231)

* Add trove classifiers for Armenian and Yiddish which have now been registered
  with PyPI.  Thanks to Henry Schreiner and Dmitry Shachnev.  (#228)

* Update documented details of Python 2 support in old versions.

Snowball 3.0.0 (2025-05-08)
===========================

Ada
---

* Bug fixes:

  + Fix invalid Ada code generated for Snowball `loop` (it was partly Pascal!)
    None of the stemmers shipped in previous releases triggered this bug, but
    the Turkish stemmer now does.

  + The Ada runtime was not tracking the current length of the string
    but instead used the current limit value or some other substitute, which
    manifested as various incorrect behaviours for code inside of `setlimit`.

  + `size` was incorrectly returning the difference between the limit and the
    backwards limit.

  + `lenof` or `sizeof` on a string variable generated Ada code that didn't
    even compile.

  + Fix incorrect preconditions on some methods in the runtime.

  + Fix bug in runtime code used by `attach`, `insert`, `<-` and string
    variable assignment when a (sub)string was replaced with a larger string.
    This bug was triggered by code in the Kraaij-Pohlmann Dutch stemmer
    implementation (which was previously not enabled by default but is now the
    standard Dutch stemmer).

  + Fix invalid code generated for `insert`, `<-` and string variable
    assignment.  This bug was triggered by code in the Kraaij-Pohlmann
    Dutch stemmer implementation (which was previously not enabled by default
    but is now the standard Dutch stemmer).

  + Generate valid code for programs which don't use `among`.  This didn't
    affect code generation for any algorithms we currently ship.

  + If the end of a routine was unreachable code the Snowball compiler
    would think the start of the next routine was also unreachable and would
    not generate it.  This didn't affect code generation for any algorithms we
    currently ship.

* Code quality:

  + Only declare variables A and C when each is needed.

  + Fix indentation of generated declarations.

  + Drop extra blank line before `Result := True`.

C/C++
-----

* Bug fixes:

  + Fix potential NULL dereference in runtime code if we failed to allocate
    memory for the p or S member for a Snowball program which uses one or more
    string variables.  Problem was introduced in Snowball 2.0.0.  Fixes #206,
    reported by Maxim Korotkov.

  + Fix invalid C code generated when a failure is handled in a context with
    the opposite direction to where it happened, for example:

        externals (stem)
        define stem as ( try backwards 'x' )

    This was fixed by changing the C generator to work like all the other
    generators and pre-generate the code to handle failure.

  + Eliminate assumptions that NULL has all-zero bit pattern.  We don't know
    of any current platforms where this assumption fails, but the C standard
    doesn't require an all-zero bit pattern for NULL.  Fixes #207.

* Optimisations:

  + Store index delta for among substring_i field.  This makes trying
    substrings after a failed match slightly faster because we can just add
    the offset to the pointer we already have to the current element.

* Code quality:

  + Improve formatting of generated code.

C#
--

* Bug fixes:

  + Add missing runtime support for testing for a string var at the current
    position when working forwards.  This situation isn't exercised by any of
    the stemming algorithms we currently ship.

  + Adjust generated code to work around a code flow analysis bug in the `mcs`
    C# compiler.

* Code quality:

  + Prune unused `using System.Text;`.

  + Generate C# with UTF-8 source encoding.  This makes the generated code
    easier to follow, which helps during development.  It's also a bit smaller.
    For now codepoints U+0590 and above are still emitted as escape sequences
    to avoid confusing source code rendering when LTR scripts are involved.

Go
--

* Optimisations:

  + Drop some unneeded Go code generated for string `$`.  None of the shipped
    stemmers use string `$`, though the Schinke Latin stemmer algorithm on the
    website does.

* Code quality:

  + Dispatch among result with `switch` instead of an `if` ... `else if` chain
    (which looks like we did because the Go generator evolved from the Python
    generator and Python didn't used to have a switch-like construct.  This
    doesn't make a measurable speed difference so it seems the Go compiler is
    optimising both to equivalent code, but using a switch here seems clearer,
    a better match for the intent, and is a bit simpler to generate.

  + Generate Go with UTF-8 source encoding.  This makes the generated code
    easier to follow, which helps during development.  It's also a bit smaller.
    For now codepoints U+0590 and above are still emitted as escape sequences
    to avoid confusing source code rendering when LTR scripts are involved.

Java
----

* The Java code generated by Snowball requires now requires Java >= 7.  Java 7
  was released in 2011, and Java 6's EOL was 2013 so we don't expect this
  to be a problematic requirement.  See #195.

* Optimisations:

  + We now store the current string in a `char[]` rather than using a
    `StringBuilder` to reduce overheads.  The `getCurrent()` method continues
    to return a Java `String`, but the `char[]` can be accessed using the new
    `getCurrentBuffer()` and `getCurrentBufferLength()` methods.  Patch from
    Robert Muir (#195).

  + Use a more efficient mechanism for calling `among` functions.  Patch from
    Robert Muir (#195).

* Code quality:

  + Consistently put `[]` right after element type for array types, which seems
    the most used style.

  + Fix javac warnings in SnowballProgram.java.

  + Improve formatting of generated code.

Javascript
----------

* Bug fixes:

  + Use base class specified by `-p` in string `$` rather than hard-coding
    `BaseStemmer` (which is the default if you don't specify `-p`).  None of
    the shipped stemmers use string `$`, though the Schinke Latin stemmer
    algorithm on the website does.

* Code quality:

  + Modernise the generated code a bit.  Loosely based on changes proposed in
    #123 by Emily Marigold Klassen.

* Other changes:

  + The Javascript runner is now specified by make variable `JSRUN` instead
    of `NODE` (since node is just one JS implementation).  The default value
    is now `node` instead of `nodejs` (older Debian and Ubuntu packages used
    `/usr/bin/nodejs` because `/usr/bin/node` was already in use by a
    completely different package, but that has since changed).

Pascal
------

* Bug fixes:

  + Add missing semicolons to code generated in some cases for a function which
    always succeeds or always fails.  The new dutch.sbl was triggering this
    bug.

  + If the end of a routine was unreachable code the Snowball compiler
    would think the start of the next routine was also unreachable and would
    not generate it.  This didn't affect code generation for any algorithms we
    currently ship.

* Code quality:

  + Eliminate commented out code generated for string `$`.  None of the shipped
    stemmers use string `$`, though the Schinke Latin stemmer algorithm on the
    website does.

* Other changes:

  + Enable warnings, etc from fpc.

  + Select GNU-style diagnostic format.

Python
------

* Optimisations:

  + Use Python set for grouping checks.  This speeds up running the Python
    testsuite by about 4%.

  + Routines used in `among` are now referenced by name directly in the
    generated code, rather than using a string containing the name.  This
    avoids a `getattr()` call each time an among wants to call a routine.  This
    doesn't seem to make a measurable speed difference, but it's cleaner and
    avoids problems with name mangling.  Suggested by David Corbett in #217.

  + Simplify code generated for `loop`.  If the iteration count is constant and
    at most 4 then iterate over a tuple which microbenchmarking shows is
    faster.  The only current uses of loop in the shipped stemmers are `loop 2`
    so benefit from this.  Otherwise we now use `range(AE)` instead of
    `range (AE, 0, -1)` (the actual value of the loop variable is never
    used so only the number of iterations matter).

* Bug fixes:

  + Correctly handle stemmer names with an underscore.

* Code quality:

  + Generate Python with UTF-8 source encoding.  This makes the generated code
    easier to follow, which helps during development.  It's also a bit smaller.
    For now codepoints U+0590 and above are still emitted as escape sequences
    to avoid confusing source code rendering when LTR scripts are involved.

* Other changes:

  + Set python_requires to indicate to install tools that the generated code
    won't work with Python 3.0.x, 3.1.x and 3.2.x (due to use of `u"foo"`
    string literals).  Closes #192 and #191, opened by Andreas Maier.

  + Add classifiers to indicate support for Python 3.3 and for 3.8 to 3.13.
    Fixes #158, reported by Dmitry Shachnev.

  + Stop marking the wheel as universal, which had started to give a warning
    message.  Patch from Dmitry Shachnev (#210).

  + Stop calling `setup.py` directly which is deprecated and now produces a
    warning - use the `build` module instead.  Patch from Dmitry Shachnev
    (#210).

Rust
----

* Optimisations:

  + Shortcut unnecessary calls to find_among, porting an optimization from the
    C generator.  In some stemming benchmarks this improves the performance
    of the rust english stemmer by about 27%.  Patch from jedav (#202).

* Code quality:

  + Suppress unused_parens warning, for example triggered by the code generated
    for `$x = x*x` (where `x` is an integer).

  + Dispatch `among` result with `match` instead of an `if` ... `else if` chain
    (which looks like we did because the Rust generator evolved from the Python
    generator and Python didn't used to have a switch-like construct.  This
    results in a 3% speed-up for an unoptimised Rust compile but doesn't seem
    to make a measurable difference when optimising so it seems the Rust
    compiler is optimising both to equivalent code.  However using a `match`
    here seems clearer, a better match for the intent, and is a bit simpler to
    generate.

  + Generate Rust with UTF-8 source encoding.  This makes the generated code
    easier to follow, which helps during development.  It's also a bit smaller.
    For now codepoints U+0590 and above are still emitted as escape sequences
    to avoid confusing source code rendering when LTR scripts are involved.

New stemming algorithms
-----------------------

* Add Esperanto stemmer from David Corbett (#185).

* Add Estonian algorithm from Linda Freienthal (#108).

Behavioural changes to existing algorithms
------------------------------------------

* Dutch: Switch to Kraaij-Pohlmann as the default for Dutch.  In case you
  want Martin Porter's Dutch stemming algorithm for compatibility, this is now
  available as `dutch_porter`.  Fixes #1, reported by gboer.

* Dutch (Kraaij-Pohlmann): Fix differences between the Snowball implementation
  and the original C implementation.

* Dutch (Kraaij-Pohlmann): Add a small number of exceptions to the Snowball
  implementation to avoid unwanted conflations.  This addresses all cases so
  far identified which Martin's Dutch stemmer handled better.  Fixes #208.

* Dutch (Porter): The "at least 3 characters" part of the R1 definition was
  actually implemented such that when working in UTF-8 it was "at least 3
  bytes".  We stripped accents normally found in Dutch except for `è` before
  setting R1, and no Dutch words starting `è` seem to stem differently
  depending on encoding, but proper nouns and other words of foreign origin may
  contain other accented characters and it seems better for the stemmer to
  handle such words the same way regardless of the encoding in use.

* English: Replace '-ogist' with '-og' to conflate "geologist" and "geology",
  etc.  Suggested by Marc Schipperheijn on snowball-discuss.

* English: Add extra condition to undoubling.  We no longer undouble if the
  double consonant is preceded by exactly "a", "e" or "o" to avoid conflating
  "add"/"ad", "egg"/"eg", "off"/"of", etc.  Fixes #182, reported by Ed Page.

* English: Avoid conflating 'emerge' and 'emergency'.  Reported by Frederick
  Ross on snowball-discuss.

* English: Avoid conflating 'evening' and 'even'.  Reported by Ann B on
  snowball-discuss.

* English: Avoid conflating 'lateral' and 'later'.  Reported by Steve Tolkin on
  snowball-discuss.

* English: Avoid conflating 'organ', 'organic' and 'organize'.

* English: Avoid conflating 'past' and 'paste'.  Reported by Sonny on
  snowball-discuss.

* English: Avoid conflating 'universe', 'universal' and 'university'.  Reported
  by Clem Wang on snowball-discuss.

* English: Handle -eed and -ing exceptions in their respective rules.
  This avoids the overhead of checking for them for the majority of
  words which don't end -eed or -ing.  It also allows us to easily handle
  vying->vie and hying->hie at basically no extra cost.  Reduces the time to
  stem all words in our English word list by nearly 2%.

* French: Remove elisions as first step.  See #187.  Originally reported by
  Paul Rudin and kelson42.

* French: Remove -aise and -aises so for example, "française" and "françaises"
  are now conflated with "français".  Fixes #209.  Originally reported by
  ririsoft and Fred Fung.

* French: Avoid incorrect conflation of `mauvais` (bad) with `mauve` (mauve,
  mallow or seagull); avoid conflating `mal` with `malais`, `pal` with
  `palais`, etc.

* French: Avoid conflating `ni` (neither/nor) with `niais`
  (inexperienced/silly) and `nie`/`nié`/`nier`/`nierais`/`nierons` (to deny).

* French: -oux -> -ou.  Fixes #91, reported by merwok.

* German: Replace with the "german2" variant.  This normalises umlauts ("ä" to
  "ae", "ö" to "oe", "ü" to "ue") which is presumably much less common in
  newly created text than it once was as modern computer systems generally
  don't have the limitations which motivated this, but there will still be
  large amounts of legacy text which it seems helpful for the stemmer to
  handle without having to know to select a variant.

  On our sample German vocabulary which contains 35033 words, 77 words give
  different stems.  A significant proportion of these are foreign words, and
  some are proper nouns.  Some cases definitely seem improved, and quite a few
  are just different but effectively just change the stem for a word or group
  of words to a stem that isn't otherwise generated.  There don't seem any
  changes that are clearly worse, though there are some changes that have both
  good and bad aspects to them.

  Fixes #92, reported by jrabensc.

* German: Don't remove -em if preceded by -syst to avoid overstemming words
  ending -system.  This change means we now conflate e.g. "system" and
  "systemen".  Partly addresses #161, reported by Olga Gusenikova.

* German: Remove -erin and -erinnen suffixes which conflates singular and
  plural female versions of nouns with the male versions.  Fixes #85 and
  partly addresses #161, reported by Olga Gusenikova.

* German: Replace -ln and -lns with -l.  This improves 82 cases in the current
  sample data without making anything worse.  Tests on a larger word list look
  good too.  Partly addresses #161, reported by Olga Gusenikova.

* German: Remove -et suffix when we safely can.  Fixes #200, reported by Robert
  Frunzke.

* Greek: Fix "faulty slice operation" for input `ισαισα`.  The fix changes
  `ισα` to stem to `ισ` instead of the empty string, which seems better (and to
  be what the second paper actually says to do if read carefully).  Fixes #204,
  reported by subnix.

* Italian: Address overstemming of "divano" (sofa) which previously stemmed to
  "div", which is the stem for 'diva' (diva).  Now it is stemmed to 'divan',
  which is what its plural form 'divani' already stemmed to.  Fixes #49,
  reported by francesco.

* Norwegian: Improve stemming of words ending -ers.  Fixes #175, reported by
  Karianne Berg.

* Norwegian: Include more accented vowels - treating "ê", "ò", "ó" and "ô"
  as vowels improves the stemming of a fairly small number of words, but
  there's basically no cost to having extra vowels in the grouping, and some
  of these words are commonly used.  Fixes #218, reported by András Jankovics.

* Romanian: Fix to work with Romanian text encoded using the correct Unicode
  characters.  Romanian uses a "comma below" diacritic on letters "s" and "t"
  ("ș" and "ț").  Before Unicode these weren't easily available so Romanian
  text was written using the visually similar "cedilla" diacritic on these
  letters instead ("ş" and "ţ").  Previously our stemmer only recognised the
  latter.  Now it maps the cedilla forms to "comma below" as a first step.
  Patch from Robert Muir.

* Spanish: Handle -acion like -ación and -ucion like -ución.  It's apparently
  common to miss off accents in Spanish, and there are examples in our test
  vocabulary that these changes help.  Proposed by Damian Janowski.

* Swedish: Replace suffix "öst" with "ös" when preceded by any of 'iklnprtuv'
  rather than just 'l'.  The new rule only requires the "öst" to be in R1
  whereas previously we required all of "löst" to be.  This second tweak
  doesn't seem to affect any words ending "löst" but it conflates a few extra
  cases when combined with the expanded list of preceding letters, and seems
  more logical linguistically (since "ös" is akin to "ous" in English).  Fixes
  #152, reported by znakeeye.

* Swedish: Remove -et/-ets in cases where it helps.  Removing -et can't be done
  unconditionally because many words end in -et where this isn't a suffix.
  However it's a very common suffix so it seems worth crafting a more complex
  condition under which to remove.  Fixes #47.

* Turkish: Remove proper noun suffixes.  For example, `Türkiye'dir` ("it is
  Turkey") is now conflated with `Türkiye` ("Turkey").  Fixes #188.

* Yiddish: Avoid generating empty stem for input "גע" (not a valid word, but
  it's better to avoid an empty stem for any non-empty input).

Optimisations to existing algorithms
------------------------------------

* General change: Use `gopast` everywhere to establish R1 and R2 as it is a
  little more efficient to do so.

* Basque: Use an empty action rather than replacing the suffix with itself
  which seems clearer and is a little more efficient.

* Dutch (Porter): Optimise prelude routine.

* English: Remove unnecessary exception for `skis` as the algorithm stems
  `skis` to `ski` by itself (`skies` and `sky` do still need a special case to
  avoid conflation with `ski` though).

* Hungarian: We no longer take digraphs into account when determining where R1
  starts.  This can only make a difference to the stemming if we removed a
  suffix that started with the last character of the digraph (or with "zs" in
  the case of "dzs"), and that doesn't happen for any of the suffixes we remove
  for any valid Hungarian words.  This simplification speeds up stemming by
  ~2% on the current sample vocabulary list.  See #216.  Thanks to András
  Jankovics for confirming no Hungarian words are affected by this change.

* Lithuanian: Remove redundant R1 check.

* Nepali: Eliminate redundant check_category_2 routine.

* Tamil: Optimise by using `among` instead of long `or` chains.  The generated
  C version now takes 43% less time to processes the test vocabulary.

* Tamil: Remove many cases which can't be triggered due to being handled by
  another case.

* Tamil: Clean up some uses of `test`.

* Tamil: Make `fix_va_start` simpler and faster.

* Tamil: Localise use of `found_a_match` flag.

* Tamil: Eliminate pointless flag changes.

* Turkish: Minor optimisations.

Code clarity improvements to existing algorithms
------------------------------------------------

* Stop noting dates changes were made in comments in the code - we now maintain
  a changelog in each algorithm's description page on the website (and the
  version control history provides a finer grained view).

* Always use `insert` instead of `<+` as the named command seems clearer.

* English: Add comments documenting motivating examples for all exceptional
  cases.

* Lithuanian: Change to recommended latin stringdef codes.  Using common codes
  makes it easier to work across algorithms, but they are more mnemonic so also
  seem clearer when just considering this one algorithm.

* Serbian: Change to recommended latin stringdef codes.  Using common codes
  makes it easier to work across algorithms, but they are more mnemonic so also
  seem clearer when just considering this one algorithm.

* Turkish: Use `{sc}` for s-cedilla and `{i}` for dotless-i to match other
  uses.

Compiler
--------

* Generic code generation improvements:

  + Show Snowball source leafname in "generated" comment at start of files.

  + Add generic reachability tracking machinery.  This facilitates various new
    optimisations, so far the following have been implemented:

    - Tail-calling
    - Simpler code for calling routines which always give the same signal
    - Simpler code when a routine ends in a integer test (this also allows
      eliminating an Ada-specific codegen optimisation which did something
      similar but only for routines which consisted *entirely* of a single
      integer test.
    - Dead code reporting and removal (only in simple cases currently)

    Currently this overlaps in functionality with the existing reachability
    tracking which is implemented on a per-language basis, and only for some
    languages.  This reachability tracking was originally added for Java
    where some unreachable code is invalid and result in a compile time error,
    but then seems to have been copied for some other newer languages which
    may or may not actually need it.  The approach it uses unfortunately
    relies on correctly updating the reachability flag anywhere in the
    generator code where reachability can change which has proved to be a
    source of bugs, some unfixed.  This new approach seems better and with some
    more work should allow us to eliminate the older code.  Fixes #83.

  + Omit check for `among` failing in generated code when we can tell at
    compile time that it can't fail.

  + Optimise `goto`/`gopast` applied to a grouping or inverted grouping (which
    is by far the most common way to use `goto`/`gopast`) for all target
    languages (new for Go, Java, Javascript, Pascal and Rust).

  + We never need to restore the cursor after `not`.  If `not` turns signal `f`
    into `t` then it sets `c` back to its old position; otherwise, `not`
    signals `f` and `c` will get reset by whatever ultimately handles this `f`
    (or the program exits and the position of `c` no longer matters).  This
    slightly improves the generated code for the `english` and `porter`
    stemmers.

  + Don't generate code for undefined or unused routines.

  + Avoid generating variable names and then not actually using them.  This
    eliminates mysterious gaps in the numbering of variables in the generated
    code.

  + Eliminate `!`/`not` from integer test code by generating the inverse
    comparison operator instead for all languages, e.g. for Python we now
    generate

      if self.I_p1 >= self.I_x:

    instead of

      if not self.I_p1 < self.I_x:

    This isn't going to be faster in compiled languages with an optimiser but
    for scripting languages it may be faster, and even if not, it makes for a
    little less work when loading the script.

  + Canonicalise `hop 1` to `next` as the generated code for `next` can be
    slightly more efficient.  This will also apply to `hop` followed by a
    constant expression which Snowball can reduce to `1`.

  + Avoid trailing whitespace in generated files.

  + Fix problems with --comments option:

    - When generating C code we would segfault for code containing `atleast`,
      `hop` or integer tests.
    - Fix missing comments for some commands in some target languages.
    - Fix inconsistent formatting of comments in some target languages.
    - Comments in C are now always on their own line - previously some were
      after at the end of the line and some on their own line which made them
      harder to follow.
    - Emit comments before `among` and before routine/external definitions.

  + Simplify more cases of numeric expressions (e.g. `x * 1` to `x`).

* Improve --help output.

* Division by zero during constant folding now gives an error.

* For `hop` followed by an unexpected token (e.g. `hop hop`) we were
  already emitting a suitable error but would then segfault.

* Emit error for redefinition of a grouping.

* Improve errors for `define` of an undeclared name.  We already peek at the
  next token to decide whether to try to parse as a routine or grouping.
  Previously we parsed as a routine if it was `as`, and a grouping otherwise,
  but routine definitions are more common and a grouping can only start with
  a literal string or a name, so now we assume a routine definition with a
  missing `as` if the next token isn't valid for either.

* Suppress duplicate (or even triplicate) "unexpected" errors for the same
  token when the compiler tried to recover from the error by adjusting the
  parse stare and marking the token to be reparsed, but the same token then
  failed to parse in the new state.

* Fix NULL pointer dereference if an undefined grouping is used in the
  definition of another grouping.

* Fix mangled error for `set` or `unset` on a non-boolean:

  test.sbl:2: nameInvalid type 98 in name_of_type()

* Emit warning if `=>` is used.  The documentation of how it works doesn't
  match the implementation, and it seems it has only ever been used in the
  Schinke stemmer implementation (which assumes the implemented behaviour).
  We've updated the Schinke implementation to avoid it.  If you're using it
  in your own Snowball code please let us know.

* Improve errors for unterminated string literals.

* Fix NULL pointer dereference on invalid code such as `$x = $y`.

* If malloc fails while compiling the compiler will now report the failure
  and exit.  Previously the NULL return from malloc wasn't checked for so
  we'd typically segfault.

* `lenof` and `sizeof` applied to a string variable now mark the variable
  as used, which avoids a bogus error followed by a confusing additional
  message if this is the only use of that variable:

  lenofsizeofbug.sbl:3: warning: string 's' is set but never used
  Unhandled type of dead assignment via sizeof

  This is situation is unlikely to occur in real world code.

* The reported line number for "string not terminated" error was one too high
  in the case where we were in a stringdef (but correct if we weren't).

* Eliminate special handling for among starter.  We now convert the starter
  to be a command before the among, adding an explicit substring if there
  isn't one.

* We now warn if the body of a `repeat` or `atleast` loop always signals
  `t` (meaning it will loop forever which is very undesirable for a stemming
  algorithm) or always signals `f` (meaning it will never loop, which seems
  unlikely to be what was intended).

* Release memory in compiler before exit.  The OS will free all allocated
  memory when a process exits, so this memory isn't actually leaked, but it can
  be annoying with when using snowball as part of a larger build process with
  some leak-finding tools.  Patch from jsteemann in #166.

* Store textual data more efficiently in memory during Snowball compilation.
  Previously almost all textual data was stored as 16 bit values, but most
  such data only uses 8 bit character values.  Doubling the memory usage
  isn't really an issue as Snowball programs are tiny, but this also
  complicated code handling such data.  Now only literal strings use the
  16 bit values.

* Fix clang -Wunused-but-set-variable warning in compiler code.

* Fix a few -Wshadow warnings in compiler and enable this warning by default.

* Tighten parsing of `writef()` format strings.  We now error out on
  unrecognised escape codes or if a numbered escape is used with too high a
  number or a non-digit.  This change reveals that the Go and Rust generators
  were using invalid escape ~A - the old writef() code was substituting this
  with just A which is what is wanted so this case was harmless but being
  lenient here could hide bugs, especially when copying code between
  generators as they don't all support the same set of format codes.

Build system
------------

* Turn on Java warnings and make them errors.

* Compile C code with -g by default.  This makes debugging easier, and
  matches the default for at least some other build systems (e.g. autotools).

* Fix "make clean" to remove all built Ada files.

* Clean `stemtest` too.  Patch from Stefano Rivera.

* Add missing `COMMON_FILES` dependency to dist targets.

* GNUmakefile: Tidy up and make more consistent

* GNUmakefile: Make use of $* to improve speed and readability.

* Use $(patsubst ...) instead of sed in .java.class rule which gives cleaner
  make output and is a bit more efficient.

* Add `WERROR` make variable to provide a way to add `-Werror` to existing
  CFLAGS.

Testsuite
---------

* Give a clear error if snowball-data isn't found.  Fixes #196, reported by
  Andrea Maccis.

* Handle not thinning testdata better.  If THIN_FACTOR is set to 1 we no longer
  run gzipped test data through awk.  We also now handle THIN_FACTOR being set
  empty as equivalent to 1 for convenience.

* csharp_stemwords: Correctly handle a stemmer name containing an underscore.

* csharp_stemwords: Make `-i` option optional and read from stdin if omitted,
  like the C version does.

* csharp_stemwords: Process the input line by line which is more helpful for
  interactive testing, and also a little faster.

* Fix Java TestApp to allow a single argument.  The documented command line
  syntax is that you only need to specify the language and there was already
  code to read from stdin if no input file was specified, but at least two
  command line options were required.

* Fix deprecation warning in TestApp.java.

* Optimise TestApp.java by creating fewer objects.  Patch from Robert Muir.

* stemwords.py: We no longer create an empty output file if we fail to open the
  input file.

* stemwords: Improve error message to say "Out of memory or internal error"
  rather than just "Out of memory".

Documentation
-------------

* Include "what is stemming" section in each README.

* Include section on threads in each README.  Based on patch for Python from
  dbcerigo.

* Document that input should be lowercase with composed accents.  See #186,
  reported by 1993fpale.

* Add README section on building, including notes on cross-compiling.  Fixes
  #205, reported by sin-ack.

* CONTRIBUTING.rst: Clarify which charsets to list

* CONTRIBUTING.rst: Add general advice section.  In particular, note to use
  spaces-only for indentation in most cases.  Thanks to Dmitry Shachnev for
  raising this point.

* CONTRIBUTING.rst: Note that UTF-8 is OK in comments.  Thanks to Dmitry
  Shachnev for asking.

* Fix some typos.  Patch from Josh Soref.

* Document that our CI now uses github actions.

* Update link to Greek stemmer PDF.  Patch from Michael Bissett (#33).

Snowball 2.2.0 (2021-11-10)
===========================

New Code Generators
-------------------

* Add Ada generator from Stephane Carrez (#135).

Javascript
----------

* Fix generated code to use integer division rather than floating point
  division.

  Noted by David Corbett.

Pascal
------

* Fix code generated for division.  Previously real division was used and the
  generated code would fail to compile with an "Incompatible types" error.

  Noted by David Corbett.

* Fix code generated for Snowball's `minint` and `maxint` constant.

Python
------

* Python 2 is no longer actively supported, as proposed on the mailing list:
  https://lists.tartarus.org/pipermail/snowball-discuss/2021-August/001721.html

* Fix code generated for division.  Previously the Python code we generated
  used integer division but rounded negative fractions towards negative
  infinity rather than zero under Python 2, and under Python 3 used floating
  point division.

  Noted by David Corbett.

Code quality Improvements
-------------------------

* C/C++: Generate INT_MIN and INT_MAX directly, including <limits.h> from
  the generated C file if necessary, and remove the MAXINT and MININT macros
  from runtime/header.h.

* C#: An `among` without functions is now generated as `static` and groupings
  are now generated as constant.  Patches from James Turner in #146 and #147.

Code generation improvements
----------------------------

* General:

  + Constant numeric subexpressions and constant numeric tests are now
    evaluated at Snowball compile time.

  + Simplify the following degenerate `loop` and `atleast` constructs where
    N is a compile-time constant:

    - loop N C where N <= 0 is a no-op.

    - loop N C where N == 1 is just C.

    - atleast N C where N <= 0 is just repeat C.

    If the value of N doesn't depend on the current target language, platform
    or Unicode settings then we also issue a warning.

Behavioural changes to existing algorithms
------------------------------------------

* german2: Fix handling of `qu` to match algorithm description.  Previously
  the implementation erroneously did `skip 2` after `qu`.  We suspect this was
  intended to skip the `qu` but that's already been done by the substring/among
  matching, so it actually skips an extra two characters.

  The implementation has always differed in this way, but there's no good
  reason to skip two extra characters here so overall it seems best to change
  the code to match the description.  This change only affects the stemming of
  a single word in the sample vocabulary - `quae` which seems to actually be
  Latin rather than German.

Optimisations to existing algorithms
------------------------------------

* arabic: Handle exception cases in the among they're exceptions to.

* greek: Remove unused slice setting, handle exception cases in the among
  they're exceptions to, and turn `substring ... among ...  or substring ...
  among ...` into a single `substring ... among ...` in cases where it is
  trivial to do so.

* hindi: Eliminate the need for variable `p`.

* irish: Minor optimisation in setting `pV` and `p1`.

* yiddish: Make use of `among` more.

Compiler
--------

* Fix handling of `len` and `lenof` being declared as names.

  For compatibility with programs written for older Snowball versions
  len and lenof stop being tokens if declared as names.  However this
  code didn't work correctly if the tokeniser's name buffer needed to
  be enlarged to hold the token name (i.e. 3 or 5 elements respectively).

* Report a clearer error if `=` is used instead of `==` in an integer test.

* Replace a single entry command list with its contents in the internal syntax
  tree.  This puts things in a more canonical form, which helps subsequent
  optimisations.

Build system
------------

* Support building on Microsoft Windows (using mingw+msys or a similar
  Unix-like environment).  Patch from Jannick in #129.

* Split out INCLUDES from CPPFLAGS so that CPPFLAGS can now be overridden by
  the user if required.  Fixes #148, reported by Dominique Leuenberger.

* Regenerate algorithms.mk only when needed rather than on every `make` run.

libstemmer
----------

* The libstemmer static library now has a `.a` extension, rather than `.o`.
  Patch from Michal Vasilek in #150.

Testsuite
---------

* stemtest: Test that numbers and numeric codes aren't damaged by any of the
  algorithms.  Regression test for #66.  Fixes #81.

* ada: Fix ada tests to fail if output differs.  There was an extra `| head
  -300` compared to other languages, which meant that the exit code of `diff`
  was ignored.  It seems more helpful (and is more consistent) not to limit how
  many differences are shown so just drop this addition.

* go: Stop thinning testdata.  It looks like we only are because the test
  harness code was based on that for rust, which was based on that for
  javascript, which was only thinning because it was reading everything into
  memory and the larger vocabulary lists were resulting in out of memory
  issues.

* javascript: Speed up stemwords.js.  Process input line-by-line rather than
  reading the whole file into memory, splitting, iterating, and creating an
  array with all the output, joining and writing out a single huge string.
  This also means we can stop thinning the test data for javascript, which we
  were only doing because the huge arabic test data file was causing out of
  memory errors.  Also drop the -p option, which isn't useful here and
  complicates the code.

* rust: Turn on optimisation in the makefile rather than the CI config.  This
  makes the tests run in about 1/5 of the time and there's really no reason to
  be thinning the testdata for rust.

Documentation
-------------

* CONTRIBUTING.rst: Improve documentation for adding a new stemming algorithm.

* Improve wording of Python docs.

Snowball 2.1.0 (2021-01-21)
===========================

C/C++
-----

* Fix decoding of 4-byte UTF-8 sequences in `grouping` checks.  This bug
  affected Unicode codepoints U+40000 to U+7FFFF and U+C0000 to U+FFFFF and
  doesn't affect any of the stemming algorithms we currently ship (#138,
  reported by Stephane Carrez).

Python
------

* Fix snowballstemmer.algorithms() method (#132, reported by kkaiser).

* Update code to generate trove language classifiers for PyPI.  All the
  natural languages we previously had stemmers for have now been added to
  PyPI's list, but Armenian and Yiddish aren't on it.  Patch from Dmitry
  Shachnev.

Code Quality Improvements
-------------------------

* Suppress GCC warning in compiler code.

* Use `const` pointers more in C runtime.

* Only use spaces for indentation in javascript code.  Change proposed by Emily
  Marigold Klassen in #123, and seems to be the modern Javascript norm.

New Snowball Language Features
------------------------------

* `lenof` and `sizeof` can now be applied to a literal string, which can be
  useful if you want to do calculations on cursor values.

  This change actually simplifies the language a little, since you can now use
  a literal string in any read-only context which accepts a string variable.

Code generation improvements
----------------------------

* General:

  + Fix bugs in the code generated to handle failure of `goto`, `gopast` or
    `try` inside `setlimit` or string-`$`.  This affected all languages (though
    the issue with `try` wasn't present for C).  These bugs don't affect any of
    the stemming algorithms we currently ship.  Reported by Stefan Petkovic on
    snowball-discuss.

  + Change `hop` with a negative argument to work as documented.  The manual
    says a negative argument to hop will raise signal f, but the implementation
    for all languages was actually to move the cursor in the opposite direction
    to `hop` with a positive argument.  The implemented behaviour is
    problematic as it allows invalidating implicitly saved cursor values by
    modifying the string outside the current region, so we've decided it's best
    to fix the implementation to match the documentation.

    The only Snowball code we're aware of which relies on this was the original
    version of the new Yiddish stemming algorithm, which has been updated not
    to rely on this.

    The compiler now issues a warning for `hop` with a constant negative
    argument (internally now converted to `false`), and for `hop` with a
    constant zero argument (internally now converted to `true`).

  + Canonicalise `among` actions equivalent to `()` such as `(true)` which
    previously resulted in an extra case in the among, and for Python
    we'd generate invalid Python code (`if` or `elif` with an empty body).
    Bug revealed by Assaf Urieli's Yiddish stemmer in #137.

  + Eliminate variables whose values are never used - they no longer have
    corresponding member variables, etc, and no code is generated for any
    assignments to them.

  + Don't generate anything for an unused `grouping`.

  + Stop warning "grouping X defined but not used" for a `grouping` which is
    only used to define another `grouping`.

* C/C++:

  + Store booleans in same array as integers.  This means each boolean is
    stored as an int instead of an unsigned char which means 4 bytes instead of
    1, but we save a pointer (4 or 8 bytes) in struct SN_env which is a win for
    all the current stemmers.  For an algorithm which uses both integers and
    booleans, we also save the overhead of allocating a block on the heap, and
    potentially improve data locality.

  + Eliminate duplicate generated C comment for sliceto.

* Pascal:

  + Avoid generating unused variables.  The Pascal code generated for the
    stemmers we ship is now warning free (tested with fpc 3.2.0).

  + Don't emit empty `private` sections.  Cosmetic, but makes the generated
    code a bit easier to follow.

* Python:

  + End `if`-chain with `else` where possible, avoiding a redundant test
    of the variable being switched on.  This optimisation kicks in for an
    `among` where all cases have commands.  This change seems to speed up `make
    check_python_arabic` by a few percent.

New stemming algorithms
-----------------------

* Add Serbian stemmer from stef4np (#113).

* Add Yiddish stemmer from Assaf Urieli (#137).

* Add Armenian stemmer from Astghik Mkrtchyan.  It's been on the website for
  over a decade, and included in Xapian for over 9 years without any negative
  feedback.

Optimisations to existing algorithms
------------------------------------

* kraaij_pohlmann: Use `$v = limit` instead of `do (tolimit setmark v)` since
  this generates simpler code, and also matches the code other algorithm
  implementations use.

  Probably for languages like C with optimising compilers the compiler
  will generate equivalent code anyway, but e.g. for Python this should be
  an improvement.

Code clarity improvements to existing algorithms
------------------------------------------------

* hindi.sbl: Fix comment typo.

Compiler
--------

* Don't count `$x = x + 1` as initialising or using `x`, so it's now handled
  like `$x += 1` already is.

* Comments are now only included in the generated code if command line option
  -comments is specified.

  The comments in the generated code are useful if you're trying to debug the
  compiler, and perhaps also if you are trying to debug your Snowball code, but
  for everyone else they just bloat the code which as the number of languages
  we support grows becomes more of an issue.

* `-parentclassname` is not only for java and csharp so don't disable it if
  those backends are disabled.

* `-syntax` now reports the value for each numeric literal.

* Report location for excessive get nesting error.

* Internally the compiler now represents negated literal numbers as a simple
  `c_number` rather than `c_neg` applied to a `c_number` with a positive value.
  This simplifies optimisations that want to check for a constant numeric
  expression.

Build system
------------

* Link binaries with LDFLAGS if it's set, which is needed for some platform
  (e.g. OpenEmbedded).  Patch from Andreas Müller (#120).

* Add missing dependencies of algorithms.go rule.

Testsuite
---------

* C: Add stemtest for low-level regression tests.

Documentation
-------------

* Document a C99 compiler as a requirement for building the snowball compiler
  (but the C code it generates should still work with any ISO C compiler).

  A few declarations mixed with code crept in some time ago (which nobody's
  complained about), so this is really just formally documenting a requirement
  which already existed.

* README: Explain what Snowball is and what Stemming is (#131, reported by Sean
  Kelly).

* CONTRIBUTING.rst: Expand section on adding a new generator.

* For Python snowballstemmer module include global NEWS instead of
  Python-specific CHANGES.rst and use README.rst as the long description.
  Patch from Dmitry Shachnev (#119).

* COPYING: Update and incorporate Python backend licensing information which
  was previously in a separate file.

Snowball 2.0.0 (2019-10-02)
===========================

C/C++
-----

* Fully handle 4-byte UTF-8 sequences.  Previously `hop` and `next` handled
  sequences of any length, but commands which look at the character value only
  handled sequences up to length 3.  Fixes #89.

* Fix handling of a 3-byte UTF-8 sequence in a grouping in `backwardmode`.

Java
----

* TestApp.java:

  - Always use UTF-8 for I/O.  Patch from David Corbett (#80).

  - Allow reading input from stdin.

  - Remove rather pointless "stem n times" feature.

  - Only lower case ASCII to match stemwords.c.

  - Stem empty lines too to match stemwords.c.

Code Quality Improvements
-------------------------

* Fix various warnings from newer compilers.

* Improve use of `const`.

* Share common functions between compiler backends rather than having multiple
  copies of the same code.

* Assorted code clean-up.

* Initialise line_labelled member of struct generator to 0.  Previously we were
  invoking undefined behaviour, though in practice it'll be zero initialised on
  most platforms.

New Code Generators
-------------------

* Add Python generator (#24).  Originally written by Yoshiki Shibukawa, with
  additional updates by Dmitry Shachnev.

* Add Javascript generator.  Based on JSX generator (#26) written by Yoshiki
  Shibukawa.

* Add Rust generator from Jakob Demler (#51).

* Add Go generator from Marty Schoch (#57).

* Add C# generator.  Based on patch from Cesar Souza (#16, #17).

* Add Pascal generator.  Based on Delphi backend from stemming.zip file on old
  website (#75).

New Snowball Language Features
------------------------------

* Add `len` and `lenof` to measure Unicode length.  These are similar to `size`
  and `sizeof` (respectively), but `size` and `sizeof` return the length in
  bytes under `-utf8`, whereas these new commands give the same result whether
  using `-utf8`, `-widechars` or neither (but under `-utf8` they are O(n) in
  the length of the string).  For compatibility with existing code which might
  use these as variable or function names, they stop being treated as tokens if
  declared to be a variable or function.

* New `{U+1234}` stringdef notation for Unicode codepoints.

* More versatile integer tests.  Now you can compare any two arithmetic
  expressions with a relational operator in parentheses after the `$`, so for
  example `$(len > 3)` can now be used when previously a temporary variable was
  required: `$tmp = len $tmp > 3`

Code generation improvements
----------------------------

* General:

  + Avoid unnecessarily saving and restoring of the cursor for more commands -
    `atlimit`, `do`, `set` and `unset` all leave the cursor alone or always
    restore its value, and for C `booltest` (which other languages already
    handled).

  + Special case handling for `setlimit tomark AE`.  All uses of setlimit in
    the current stemmers we ship follow this pattern, and by special-casing we
    can avoid having to save and restore the cursor (#74).

  + Merge duplicate actions in the same `among`.  This reduces the size of the
    switch/if-chain in the generated code which dispatch the among for many of
    the stemmers.

  + Generate simpler code for `among`.  We always check for a zero return value
    when we call the among, so there's no point also checking for that in the
    switch/if-chain.  We can also avoid the switch/if-chain entirely when
    there's only one possible outcome (besides the zero return).

  + Optimise code generated for `do <function call>`.  This speeds up "make
    check_python" by about 2%, and should speed up other interpreted languages
    too (#110).

  + Generate more and better comments referencing snowball source.

  + Add homepage URL and compiler version as comments in generated files.

* C/C++:

  + Fix `size` and `sizeof` to not report one too high (reported by Assem
    Chelli in #32).

  + If signal `f` from a function call would lead to return from the current
    function then handle this and bailing out on an error together with a
    simple `if (ret <= 0) return ret;`

  + Inline testing for a single character literals.

  + Avoiding generating `|| 0` in corner case - this can result in a compiler
    warning when building the generated code.

  + Implement `insert_v()` in terms of `insert_s()`.

  + Add conditional `extern "C"` so `runtime/api.h` can be included from C++
    code.  Closes #90, reported by vvarma.

* Java:

  + Fix functions in `among` to work in Java.  We seem to need to make the
    methods called from among `public` instead of `private`, and to call them
    on `this` instead of the `methodObject` (which is cleaner anyway).  No
    revision in version control seems to generate working code for this case,
    but Richard says it definitely used to work - possibly older JVMs failed to
    correctly enforce the access controls when methods were invoked by
    reflection.

  + Code after handling `f` by returning from the current function is
    unreachable too.

  + Previously we incorrectly decided that code after an `or` was
    unreachable in certain cases.  None of the current stemmers in the
    distribution triggered this, but Martin Porter's snowball version
    of the Schinke Latin stemmer does.  Fixes #58, reported by Alexander
    Myltsev.

  + The reachability logic was failing to consider reachability from
    the final command in an `or`.  Fixes #82, reported by David Corbett.

  + Fix `maxint` and `minint`.  Patch from David Corbett in #31.

  + Fix `$` on strings.  The previous generated code was just wrong.  This
    doesn't affect any of the included algorithms, but for example breaks
    Martin Porter's snowball implementation of Schinke's Latin Stemmer.
    Issue noted by Jakob Demler while working on the Rust backend in #51,
    and reported in the Schinke's Latin Stemmer by Alexander Myltsev
    in #58.

  + Make SnowballProgram objects serializable.  Patch from Oleg Smirnov in #43.

  + Eliminate range-check implementation for groupings.  This was removed from
    the C generator 10 years earlier, isn't used for any of the existing
    algorithms, and it doesn't seem likely it would be - the grouping would
    have to consist entirely of a contiguous block of Unicode code-points.

  + Simplify code generated for `repeat` and `atleast`.

  + Eliminate unused return values and variables from runtime functions.

  + Only import the `among` and `SnowballProgram` classes if they're actually
    used.

  + Only generate `copy_from()` method if it's used.

  + Merge runtime functions `eq_s` and `eq_v` functions.

  + Java arrays know their own length so stop storing it separately.

  + Escape char 127 (DEL) in generated Java code.  It's unlikely that this
    character would actually be used in a real stemmer, so this was more of a
    theoretical bug.

  + Drop unused import of InvocationTargetException from SnowballStemmer.
    Reported by GerritDeMeulder in #72.

  + Fix lint check issues in generated Java code.  The stemmer classes are only
    referenced in the example app via reflection, so add
    @SuppressWarnings("unused") for them.  The stemmer classes override
    equals() and hashCode() methods from the standard java Object class, so
    mark these with @Override.  Both suggested by GerritDeMeulder in #72.

  + Declare Java variables at point of use in generated code.  Putting all
    declarations at the top of the function was adding unnecessary complexity
    to the Java generator code for no benefit.

  + Improve formatting of generated code.

New stemming algorithms
-----------------------

* Add Tamil stemmer from Damodharan Rajalingam (#2, #3).

* Add Arabic stemmer from Assem Chelli (#32, #50).

* Add Irish stemmer from Jim O'Regan (#48).

* Add Nepali stemmer from Arthur Zakirov (#70).

* Add Indonesian stemmer from Olly Betts (#71).

* Add Hindi stemmer from Olly Betts (#73). Thanks to David Corbett for review.

* Add Lithuanian stemmer from Dainius Jocas (#22, #76).

* Add Greek stemmer from Oleg Smirnov (#44).

* Add Catalan and Basque stemmers from Israel Olalla (#104).

Behavioural changes to existing algorithms
------------------------------------------

* Portuguese:

  + Replace incorrect Spanish suffixes by Portuguese suffixes (#1).

* French:

  + The MSDOS CP850 version of the French algorithm was missing changes present
    in the ISO8859-1 and Unicode versions.  There's now a single version of
    each algorithm which was based on the Unicode version.

  + Recognize French suffixes even when they begin with diaereses.  Patch from
    David Corbett in #78.

* Russian:

  + We now normalise 'ё' to 'е' before stemming.  The documentation has long
    said "we assume ['ё'] is mapped into ['е']" but it's more convenient for
    the stemmer to actually perform this normalisation.  This change has no
    effect if the caller is already normalising as we recommend.  It's a change
    in behaviour they aren't, but 'ё' occurs rarely (there are currently no
    instances in our test vocabulary) and this improves behaviour when it does
    occur.  Patch from Eugene Mirotin (#65, #68).

* Finish:

  + Adjust the Finnish algorithm not to mangle numbers.  This change also
    means it tends to leave foreign words alone.  Fixes #66.

* Danish:

  + Adjust Danish algorithm not to mangle alphanumeric codes. In particular
    alphanumeric codes ending in a double digit (e.g. 0x0e00, hal9000,
    space1999) are no longer mangled.  See #81.

Optimisations to existing algorithms
------------------------------------

* Turkish:

  + Simplify uses of `test` in stemmer code.

  + Check for 'ad' or 'soyad' more efficiently, and without needing the
    strlen variable.  This speeds up "make check_utf8_turkish" by 11%
    on x86 Linux.

* Kraaij-Pohlmann:

  + Eliminate variable x `$p1 <= cursor` is simpler and a little more efficient
    than `setmark x $x >= p1`.

Code clarity improvements to existing algorithms
------------------------------------------------

* Turkish:

  + Use , for cedilla to match the conventions used in other stemmers.

* Kraaij-Pohlmann:

  + Avoid cryptic `[among ( (])` ... `)` construct - instead use the same
    `[substring] among (` ... `)` construct we do in other stemmers.

Compiler
--------

* Support conventional --help and --version options.

* Warn if -r or -ep used with backend other than C/C++.

* Warn if encoding command line options are specified when generating code in a
  language with a fixed encoding.

* The default classname is now set based on the output filename, so `-n` is now
  often no longer needed.  Fixes #64.

* Avoid potential one byte buffer over-read when parsing snowball code.

* Avoid comparing with uninitialised array element during compilation.

* Improve `-syntax` output for `setlimit L for C`.

* Optimise away double negation so generators don't have to worry about
  generating `--` (decrement operator in many languages).  Fixes #52, reported
  by David Corbett.

* Improved compiler error and warning messages:

  - We now report FILE:LINE: before each diagnostic message.

  - Improve warnings for unused declarations/definitions.

  - Warn for variables which are used, but either never initialised
    or never read.

  - Flag non-ASCII literal strings.  This is an error for wide Unicode, but
    only a warning for single-byte and UTF-8 which work so long as the source
    encoding matches the encoding used in the generated stemmer code.

  - Improve error recovery after an undeclared `define`.  We now sniff the
    token after the identifier and if it is `as` we parse as a routine,
    otherwise we parse as a grouping.  Previously we always just assumed it was
    a routine, which gave a confusing second error if it was a grouping.

  - Improve error recovery after an unexpected token in `among`.  Previously
    we acted as if the unexpected token closed the `among` (this probably
    wasn't intended but just a missing `break;` in a switch statement).  Now we
    issue an error and try the next token.

* Report error instead of silently truncating character values (e.g. `hex 123`
  previously silently became byte 0x23 which is `#` rather than a
  g-with-cedilla).

* Enlarge the initial input buffer size to 8192 bytes and double each time we
  hit the end.  Snowball programs are typically a few KB in size (with the
  current largest we ship being the Greek stemmer at 27KB) so the previous
  approach of starting with a 10 byte input buffer and increasing its size by
  50% plus 40 bytes each time it filled was inefficient, needing up to 15
  reallocations to load greek.sbl.

* Identify variables only used by one `routine`/`external`.  This information
  isn't yet used, but such variables which are also always written to before
  being read can be emitted as local variables in most target languages.

* We now allow multiple source files on command line, and allow them to be
  after (or even interspersed) with options to better match modern Unix
  conventions.  Support for multiple source files allows specifying a single
  byte character set mapping via a source file of `stringdef`.

* Avoid infinite recursion in compiler when optimising a recursive snowball
  function.  Recursive functions aren't typical in snowball programs, but
  the compiler shouldn't crash for any input, especially not a valid one.
  We now simply limit on how deep the compiler will recurse and make the
  pessimistic assumption in the unlikely event we hit this limit.

Build system
------------

* `make clean` in C libstemmer_c distribution now removes `examples/*.o`.
  (#59)

* Fix all the places which previously had to have a list of stemmers to work
  dynamically or be generated, so now only modules.txt needs updating to add
  a new stemmer.

* Add check_java make target which runs tests for java.

* Support gzipped test data (the uncompressed arabic test data is too big for
  github).

* GNUmakefile: Drop useless `-eprefix` and `-r` options from snowball
  invocations for Java - these are only meaningful when generating C code.

* Pass CFLAGS when linking which matches convention (e.g. automake does it) and
  facilitates use of tools such as ASan.  Fixes #84, reported by Thomas
  Pointhuber.

* Add CI builds with -std=c90 to check compiler and generated code are C90
  (#54)

libstemmer
----------

* Split out CPPFLAGS from CFLAGS and use CFLAGS when linking stemwords.

* Add -O2 to CFLAGS.

* Make generated tables of encodings and modules const.

* Fix clang static analyzer memory leak warning (in practice this code path
  can never actually be taken).  Patch from Patrick O. Perry (#56)

Documentation
-------------

* Added copyright and licensing details (#10).

* Document that libstemmer supports ISO_8859_2 encoding.  Currently hungarian
  and romanian are available in ISO_8859_2.

* Remove documentation falsely claiming that libstemmer supports CP850
  encoding.

* CONTRIBUTING.rst: Add guidance for contributing new stemming algorithms and
  new language backends.

* Overhaul libstemmer_python_README.  Most notably, replace the benchmark data
  which was very out of date.