Anchor the utf-8-variants regexes at the true end of the buffer (fixes #236)

[sebollin]

Fixes #236.

CESU8_EXPR, SURROGATE_EXPR and NULL_EXPR use $ where they mean "the buffer ends here", but in Python $ also matches just before a trailing newline. A \n at the end of the input therefore gets consumed as part of the match:

>>> b"\xc0\n".decode("utf-8-variants")          # before
'\x00'
>>> b"\xed\xa0\xbd\xed\xb8\n".decode("utf-8-variants")
'😊'

The second one is the bad case: _buffer_decode_surrogates builds the codepoint from input[5] & 0x3F, so the newline byte becomes the low six bits and an invalid five-byte sequence decodes to U+1F60A instead of raising.

This replaces the seven $ anchors with \Z, which only matches at the very end of the string. The truncated-input behaviour the $ alternatives exist for is unchanged, since a truncated buffer ends where \Z matches.

After the change both examples raise UnicodeDecodeError, and ftfy.fix_encoding("Hi guys í ½í¸\n") keeps the newline instead of returning 'Hi guys 😊'.

Testing. The existing suite passes exactly as it does on main (336 passed, 10 xfailed under tests/; the same 7 environment-related failures appear with and without the patch on my machine — the CLI entry point and one wcwidth doctest).

I also ran a differential battery of 65,268 byte sequences — the full sweep of ED xx yy, C0 followed by every byte, six-byte CESU-8 candidates, truncations, overlong forms, valid UTF-8 mixed with CESU-8, and ~30k noise strings weighted towards 0xED/0xC0 — comparing the codec before and after. Exactly four sequences change, and all four are the bug:

b"\xc0\n"                  '\x00'    -> UnicodeDecodeError
b"\x0a\xc0\n"              '\n\x00'  -> UnicodeDecodeError
b"\xc3\xa9\xc0\n"          'é\x00'   -> UnicodeDecodeError
b"\xed\xa0\xbd\xed\xb8\n"  '😊'      -> UnicodeDecodeError

Sequences that should still decode are untouched, including a newline after a complete sequence (b"\xed\xa0\xbd\xed\xb8\x8d\n" and b"\xc0\x80\n"), and feeding the incremental decoder one byte at a time gives the same result as decoding the whole buffer.

Found while porting the codec to R for lupa, which needs CESU-8 decoding without a Python dependency.

[sebollin]

Independent verification of this PR.

The defect reproduces on main (74dd045). The trailing newline is what
makes both cases go wrong; with any other final byte they raise correctly:

>>> from ftfy import bad_codecs
>>> b'\xc0\n'.decode('utf-8-variants')
'\x00'                      # the newline is swallowed and a NUL appears
>>> b'\xc0X'.decode('utf-8-variants')
UnicodeDecodeError          # correct

>>> b'\xed\xa0\xbd\xed\xb8\n'.decode('utf-8-variants')
'😊'                        # a codepoint is invented from a truncated pair
>>> b'\xed\xa0\xbd\xed\xb8X'.decode('utf-8-variants')
UnicodeDecodeError          # correct

With this branch applied, both raise UnicodeDecodeError, which is what the
malformed input deserves.

Test suite: 357 passed, 10 xfailed — identical to main. No test
regressed.

Environment note: main already fails one unrelated doctest here,
ftfy/formatting.py::ftfy.formatting.monospaced_width, before and after the
change. Python 3.13.13.

The second case looks like the more serious of the two: silently inventing a
character from an incomplete surrogate pair is worse than the NUL, because
nothing downstream can tell it did not come from the source.