Skip to content

fix(cli): write stdout and stderr as UTF-8, not the locale encoding - #189

Merged
LukasGold merged 1 commit into
feat/mcp-serverfrom
fix/187-non-utf-8-on-windows
Sep 21, 2026
Merged

LukasGold merged 1 commit into
feat/mcp-serverfrom
fix/187-non-utf-8-on-windows

Conversation

@LukasGold

@LukasGold LukasGold commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #187.

Targets feat/mcp-server, because the CLI only exists on that branch.

Problem

Python encodes a redirected stream with the locale encoding, which is cp1252 on a German Windows system. osw --json entity get ... therefore wrote Änderungen as the byte \xc4 where UTF-8 needs \xc3\x84, and exited 0, so the failure only appeared in whatever parsed the output.

Three further facts established while fixing it:

  • The human-readable output has the same defect. Both paths go through the single typer.echo in _run.
  • A character cp1252 cannot represent at all (Japanese, Greek, Cyrillic) does not corrupt quietly. It raises UnicodeEncodeError, so the command exits 1 and writes nothing.
  • The issue names src/osw/service/serialization.py as a cause. That is not correct: its ensure_ascii=False calls only build in-memory values and never reach a stream.

Change

_force_utf8_output() in src/osw/cli/main.py reconfigures sys.stdout and sys.stderr to UTF-8. The @app.callback() calls it before any output, including the [osw] configuration banner.

  • ensure_ascii=False stays in src/osw/cli/render.py, so German labels remain readable rather than becoming \u00c4 escapes.
  • stderr is covered too, for three reasons: Context.guard redirects captured stdout to stderr under --json, error messages name the page they failed on, and the osw logger writes its records there.
  • reconfigure() changes the stream object in place instead of replacing it. The logging.StreamHandler that osw.enable_logging attaches at import time holds that same object, so it writes UTF-8 after the switch without being rebuilt. Measured, not assumed. rich also reads file.encoding per write, so it follows the switch as well.
  • errors=stream.errors is passed as well. reconfigure() resets the handler to strict otherwise, which would drop the backslashreplace Python gives stderr so that reporting a failure cannot itself raise.
  • A stream a host application substituted is skipped unless it offers both reconfigure and a non-None errors. Reading .errors off an object that lacks it raises, and passing errors=None means strict, which is the handler this function exists to preserve.
  • Scope is the CLI entry point only. osw-mcp = osw.mcp.server:main is a separate entry point and does not import osw.cli.main.

Known limitation

Click prints help and rejects an unknown root-level name before any callback runs, so those paths keep the locale encoding. Measured with PYTHONIOENCODING=cp1252 across nine invocations, two still write locale bytes: an unknown command name, and an unknown root option name, both echoed back in a usage error on stderr with exit code 2. A name typed after the command is not affected, because click resolves the command and runs the callback before parsing that command's own arguments.

Neither path carries wiki content. Every help string in the package is ASCII, now held by a test that walks the real click command tree, and rich substitutes its box-drawing characters once the stream is not UTF-8. Closing the gap entirely would mean moving the console script from osw.cli.main:app to a main() wrapper.

Not addressed, and separate from this issue: src/osw/service/params.py:34 reads --jsondata - from stdin with the locale encoding, which is the mirror-image defect on the input side.

Tests

Eight tests in tests/test_cli.py. Four drive the real CLI through CliRunner(charset="cp1252"), which gives the captured stream the locale encoding and so reproduces the Windows condition on Linux CI too. They assert on stdout_bytes / stderr_bytes, not on the already-decoded string, which click decodes with errors="replace" and would mask the defect.

  • --json output decodes as UTF-8 with a German label
  • human-readable output decodes as UTF-8 with a German label
  • an error message on stderr decodes as UTF-8 with a German page title
  • a Japanese label is written rather than raising
  • _force_utf8_output keeps the error handler each stream was given
  • a substituted stream with no usable errors value is left alone
  • a logging.StreamHandler holding sys.stderr writes UTF-8 after the switch
  • every help string in the command tree is ASCII

Each was watched failing before the code that makes it pass. Full suite: 574 passed, also with GITHUB_ACTIONS=true set, which forces the rich colour rendering CI uses.

@LukasGold LukasGold closed this Sep 21, 2026
@LukasGold LukasGold reopened this Sep 21, 2026
@LukasGold
LukasGold force-pushed the fix/187-non-utf-8-on-windows branch 2 times, most recently from 817f04c to 1fe61ac Compare September 21, 2026 10:58
@github-actions

Copy link
Copy Markdown
Contributor

Release preview

No version bump from the current commits (stays at v2.4.0). Use conventional commit types (feat, fix, ...) to trigger a release.

Changelog preview (truncated)

Preview via python-semantic-release and conventional commits.

@LukasGold
LukasGold force-pushed the fix/187-non-utf-8-on-windows branch 3 times, most recently from 7c7031d to ecd0766 Compare September 21, 2026 11:23
- a redirected stream used cp1252 on Windows, so a German label reached
  the consumer as bytes no JSON parser could read, with exit code 0
- a character cp1252 cannot represent raised UnicodeEncodeError instead
- the app callback reconfigures both streams before any output
- errors= is passed too, or reconfigure would reset stderr to strict
- --help and an unknown command name still bypass it; neither carries
  wiki content

Closes #187
@LukasGold
LukasGold force-pushed the fix/187-non-utf-8-on-windows branch from ecd0766 to 0b49e89 Compare September 21, 2026 11:24
@LukasGold
LukasGold merged commit 3a0d704 into feat/mcp-server Sep 21, 2026
11 checks passed
@LukasGold
LukasGold deleted the fix/187-non-utf-8-on-windows branch September 21, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant