fix(cli): write stdout and stderr as UTF-8, not the locale encoding - #189
Merged
Merged
Conversation
LukasGold
force-pushed
the
fix/187-non-utf-8-on-windows
branch
2 times, most recently
from
September 21, 2026 10:58
817f04c to
1fe61ac
Compare
Contributor
Release previewNo version bump from the current commits (stays at Changelog preview (truncated)Preview via python-semantic-release and conventional commits. |
LukasGold
force-pushed
the
fix/187-non-utf-8-on-windows
branch
3 times, most recently
from
September 21, 2026 11:23
7c7031d to
ecd0766
Compare
- a redirected stream used cp1252 on Windows, so a German label reached the consumer as bytes no JSON parser could read, with exit code 0 - a character cp1252 cannot represent raised UnicodeEncodeError instead - the app callback reconfigures both streams before any output - errors= is passed too, or reconfigure would reset stderr to strict - --help and an unknown command name still bypass it; neither carries wiki content Closes #187
LukasGold
force-pushed
the
fix/187-non-utf-8-on-windows
branch
from
September 21, 2026 11:24
ecd0766 to
0b49e89
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #187.
Targets
feat/mcp-server, because the CLI only exists on that branch.Problem
Python encodes a redirected stream with the locale encoding, which is cp1252 on a German Windows system.
osw --json entity get ...therefore wroteÄnderungenas the byte\xc4where UTF-8 needs\xc3\x84, and exited 0, so the failure only appeared in whatever parsed the output.Three further facts established while fixing it:
typer.echoin_run.UnicodeEncodeError, so the command exits 1 and writes nothing.src/osw/service/serialization.pyas a cause. That is not correct: itsensure_ascii=Falsecalls only build in-memory values and never reach a stream.Change
_force_utf8_output()insrc/osw/cli/main.pyreconfiguressys.stdoutandsys.stderrto UTF-8. The@app.callback()calls it before any output, including the[osw]configuration banner.ensure_ascii=Falsestays insrc/osw/cli/render.py, so German labels remain readable rather than becoming\u00c4escapes.Context.guardredirects captured stdout to stderr under--json, error messages name the page they failed on, and the osw logger writes its records there.reconfigure()changes the stream object in place instead of replacing it. Thelogging.StreamHandlerthatosw.enable_loggingattaches at import time holds that same object, so it writes UTF-8 after the switch without being rebuilt. Measured, not assumed. rich also readsfile.encodingper write, so it follows the switch as well.errors=stream.errorsis passed as well.reconfigure()resets the handler tostrictotherwise, which would drop thebackslashreplacePython gives stderr so that reporting a failure cannot itself raise.reconfigureand a non-Noneerrors. Reading.errorsoff an object that lacks it raises, and passingerrors=Nonemeansstrict, which is the handler this function exists to preserve.osw-mcp = osw.mcp.server:mainis a separate entry point and does not importosw.cli.main.Known limitation
Click prints help and rejects an unknown root-level name before any callback runs, so those paths keep the locale encoding. Measured with
PYTHONIOENCODING=cp1252across nine invocations, two still write locale bytes: an unknown command name, and an unknown root option name, both echoed back in a usage error on stderr with exit code 2. A name typed after the command is not affected, because click resolves the command and runs the callback before parsing that command's own arguments.Neither path carries wiki content. Every help string in the package is ASCII, now held by a test that walks the real click command tree, and rich substitutes its box-drawing characters once the stream is not UTF-8. Closing the gap entirely would mean moving the console script from
osw.cli.main:appto amain()wrapper.Not addressed, and separate from this issue:
src/osw/service/params.py:34reads--jsondata -from stdin with the locale encoding, which is the mirror-image defect on the input side.Tests
Eight tests in
tests/test_cli.py. Four drive the real CLI throughCliRunner(charset="cp1252"), which gives the captured stream the locale encoding and so reproduces the Windows condition on Linux CI too. They assert onstdout_bytes/stderr_bytes, not on the already-decoded string, which click decodes witherrors="replace"and would mask the defect.--jsonoutput decodes as UTF-8 with a German label_force_utf8_outputkeeps the error handler each stream was givenerrorsvalue is left alonelogging.StreamHandlerholdingsys.stderrwrites UTF-8 after the switchEach was watched failing before the code that makes it pass. Full suite: 574 passed, also with
GITHUB_ACTIONS=trueset, which forces the rich colour rendering CI uses.