fix: keep a character Markdown does not trim at the end of a line - #343
Merged
Conversation
`TRAILING_WHITESPACE_PATTERN` decides two things: what a trimmed position leaves out of the file, and what `state.safe` never sees. Matching `\s` selected every Unicode space separator, so a no-break space closing a line was deleted rather than trimmed. It predates #278, where the match only ever chose characters that were appended raw a moment later and widening it past the trimmed set changed nothing; #278 gave the match its second meaning and the extra characters became losses at that point. The character now reaches `state.safe` with the rest of the value, which is what its escapes have to be decided against rather than only what survives. A backslash before one is no longer escaped, because the character stands between it and the line ending, so it no longer spells a hard break.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A no-break space closing a line was deleted from the file on the first save. Unlike the whitespace #278 and #339 leave out, nothing about the character is trimmed on read: it renders, and it survives a reload wherever it is written, so removing it lost text rather than settling the file.
TRAILING_WHITESPACE_PATTERNdecides two things — what a trimmed position leaves out of the file, and whatstate.safenever sees. Matching\sselected every Unicode space separator rather than the space and tab CommonMark trims at a line edge, soclosesTrimmedContentdropped characters alongside the ones it was meant to. The pattern is now/[\t ]+$/u, the class the leading edge already uses and the onethematicBreakMarkdown.tsalready spells the same way.The pattern predates #278, where it only ever selected characters that were appended raw a moment later, so being wider than the trimmed set changed nothing. #278 gave the match its second meaning and the extra characters became losses at that point.
Narrowing it means the character reaches
state.safewith the rest of the value, which is what its escapes have to be decided against rather than only what survives. A backslash before one is no longer escaped: the character stands between it and the line ending, so it no longer spells a hard break. The same holds for a*,_, or~run, a bracket, a bare autolink, an email address, and awwwaddress, none of which change beyond keeping the character.Related Issue
Closes #341
Verification
markdownCompatibility.test.tsgains rows inLine-final whitespace, beside the ones asserting what that position leaves out. A no-break space is kept where it closes a paragraph, a heading, a list item, a blockquote, and a table cell, an em space and an ideographic space are kept closing a paragraph, and the reloaded document is asserted to hold the character rather than only the bytes.Three rows put the two classes against each other on one line, since that is what separates a trim from a deletion: a space the character follows is kept because the character ends the line, and a space or a tab following the character is still left out while the character stays.
Two rows assert the escapes the character now takes part in: a backslash it separates from the line ending is written bare, and a bare autolink before it stays bare.
corpus/commonmark/text-and-breaks.mdgainsTrailing whitespace a parse keeps, the contrast to the trimmed-whitespace section above it, with each invisible character named in prose as that file already does for its trailing spaces. This is what makes the class a standing guard:corpusRoundTrip.test.tscompares the document either side of a save, and reverting the one-line change makespreserves the document across a save for commonmark/text-and-breaks.mdfail. Byte convergence alone cannot see the defect, because the old behavior was stable from the first save and only lossy.pnpm check:frontendpasses. The backend is untouched, sopnpm check:backendwas not run.Not verified: the manual pass over
corpus/in the running application, which covers rendering, interaction, and navigator behavior beyond the automated round trip.Notes
A character written as
was never affected and is unchanged. It is written from the source stored on its mark and never reaches the trim, which is why the character-reference coverage showed no sign of this.The corpus fixture is not named in the issue's completion criteria. It is added because the guard was already the right home for this class and only lacked a case, at the cost of putting invisible characters into a fixture
corpusRoundTrip.test.tsandsourceProjectionEscapeCorpus.test.tsboth read.This is the third pass over the same line. #278 leaves out whitespace closing a line, #339 does the same for whitespace opening one, and the two together are what turned a pattern that was merely wider than it needed to be into one that deleted text.