fix: treat Indic conjunct clusters as a single grapheme - #3136
Merged
Conversation
🦋 Changeset detectedLatest commit: dac0376 The changes in this PR will be included in the next version bump. This PR includes changesets to release 14 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📦 Bundle Stats —
|
| Metric | Value | vs main (586c65a) |
|---|---|---|
| Internal (raw) | 814.3 KB | +18.3 KB, +2.3% |
| Internal (gzip) | 180.3 KB | +3.0 KB, +1.7% |
| Bundled (raw) | 1.43 MB | +18.3 KB, +1.3% |
| Bundled (gzip) | 350.0 KB | +3.5 KB, +1.0% |
| Import time | 76ms | -2ms, -2.3% |
@portabletext/editor/behaviors
| Metric | Value | vs main (586c65a) |
|---|---|---|
| Internal (raw) | 4.2 KB | - |
| Internal (gzip) | 1.4 KB | - |
| Bundled (raw) | 3.9 KB | - |
| Bundled (gzip) | 1.3 KB | - |
| Import time | 2ms | -0ms, -1.7% |
@portabletext/editor/plugins
| Metric | Value | vs main (586c65a) |
|---|---|---|
| Internal (raw) | 5.2 KB | - |
| Internal (gzip) | 1.8 KB | - |
| Bundled (raw) | 5.0 KB | - |
| Bundled (gzip) | 1.7 KB | - |
| Import time | 6ms | -0ms, -1.7% |
@portabletext/editor/selectors
| Metric | Value | vs main (586c65a) |
|---|---|---|
| Internal (raw) | 96.4 KB | - |
| Internal (gzip) | 22.3 KB | - |
| Bundled (raw) | 92.5 KB | - |
| Bundled (gzip) | 21.0 KB | - |
| Import time | 6ms | +0ms, +3.9% |
@portabletext/editor/traversal
| Metric | Value | vs main (586c65a) |
|---|---|---|
| Internal (raw) | 40.7 KB | - |
| Internal (gzip) | 10.6 KB | - |
| Bundled (raw) | 41.1 KB | - |
| Bundled (gzip) | 10.6 KB | - |
| Import time | 5ms | -0ms, -1.0% |
@portabletext/editor/utils
| Metric | Value | vs main (586c65a) |
|---|---|---|
| Internal (raw) | 34.1 KB | - |
| Internal (gzip) | 8.7 KB | - |
| Bundled (raw) | 32.5 KB | - |
| Bundled (gzip) | 8.5 KB | - |
| Import time | 5ms | +0ms, +1.1% |
🗺️ . · ./behaviors · ./plugins · ./selectors · ./traversal · ./utils · Artifacts
Details
- Import time regressions over 10% are flagged with
⚠️ - Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.
📦 Bundle Stats — @portabletext/markdown
Compared against main (586c65a3)
| Metric | Value | vs main (586c65a) |
|---|---|---|
| Internal (raw) | 53.0 KB | - |
| Internal (gzip) | 11.6 KB | - |
| Bundled (raw) | 351.8 KB | - |
| Bundled (gzip) | 99.4 KB | - |
| Import time | 29ms | -1ms, -4.7% |
🗺️ View treemap · Artifacts
Details
- Import time regressions over 10% are flagged with
⚠️ - Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.
christianhg
marked this pull request as ready for review
August 21, 2026 08:09
christianhg
force-pushed
the
gb9c-indic-conjunct-graphemes
branch
from
August 21, 2026 08:18
d1ee0a8 to
c4713c2
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c4713c2. Configure here.
christianhg
force-pushed
the
gb9c-indic-conjunct-graphemes
branch
from
August 21, 2026 08:26
1b674d0 to
a98bf3d
Compare
christianhg
force-pushed
the
gb9c-indic-conjunct-graphemes
branch
from
August 21, 2026 08:31
a98bf3d to
6d71db0
Compare
christianhg
force-pushed
the
gb9c-indic-conjunct-graphemes
branch
from
August 24, 2026 11:30
6d71db0 to
1dffdfa
Compare
Contributor
Bundle StatsWarning 1 significant change. @portabletext/editor🔴 All scenario measurements (7)🗺️
Significant means at least 1.0 KB and 1% gzip, or at least 5 ms and 10% import time. |
`getCharacterDistance` implemented the Unicode UAX 29 grapheme rules through GB11 and GB12/13 but predated GB9c, so an Indic conjunct cluster (consonant + virama + consonant, e.g. Devanagari क्ष) was split at the virama: caret movement and character positions treated the cluster as two characters where browsers render one glyph. Port the GB9c implementation from upstream Slate (ianstormtaylor/slate#6074): `InCB` Consonant/Extend/Linker codepoint classes, a GB9c entry in `NonBoundaryPairs`, and a deferred `endsWithConjunctLinker` check that only suppresses the boundary when the left context actually contains a conjunct linker. The character tables are identical to upstream's; they are composed into `RegExp`s from shared class strings instead of upstream's inline literals, so each table ships once (~6 KB of bundled Unicode data instead of ~18). Every consumer of `getCharacterDistance` (caret movement, positions, deletion units) now steps over the full cluster. Forward delete at the start of a cluster now removes the whole cluster. Backspace stays at one code unit per press: the whole-cluster delete composes with the compound-script reinsert in `delete-internal.ts`. That reinsert was keyed to script ranges that miss Gujarati, Tai Tham, Balinese, Sundanese, Javanese, and Meetei Mayek, which GB9c now glues, so `COMPOUND_SCRIPT_REGEX` additionally matches those scripts' BMP conjunct linkers (every GB9c cluster contains its linker). The supplementary-plane linkers are left out deliberately: the reinsert slices code units and would split a surrogate pair, so backspace on those scripts removes the whole cluster. Pinned by unit tests adapted from upstream (LTR and RTL, red on the pre-GB9c code) and browser tests: forward delete over a cluster (red pre-GB9c) and backspace one-code-unit parity for Devanagari (in-range script) and Gujarati (linker-keyed, red without the regex extension).
Backspace on an Indic conjunct cluster removes one code unit at a time (the compound-script reinsert in `delete-internal.ts`), but that override is keyed to a hardcoded set of script ranges. Pin that a ZWJ emoji sequence, which matches none of them, is removed whole by a single backspace.
christianhg
force-pushed
the
gb9c-indic-conjunct-graphemes
branch
from
August 24, 2026 11:32
1dffdfa to
dac0376
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moving the caret across an Indic conjunct cluster, for example Devanagari क्ष (consonant + virama + consonant), stopped inside the cluster: the grapheme walker in
engine/utils/string.tsimplements the UAX #29 rules through GB11 and GB12/13 but predates rule GB9c, which keeps conjunct clusters together. Browsers render the cluster as one glyph, so arrowing over it took multiple key presses and character positions disagreed with what the user sees.This ports the GB9c implementation from upstream Slate (ianstormtaylor/slate#6074): the
InCBConsonant/Extend/Linker codepoint classes, a GB9c entry in the non-boundary pair table, and a deferred check that only suppresses the boundary when the left context actually contains a conjunct linker. The ported logic and character tables are identical to upstream's, verified codepoint by codepoint; the tables are composed intoRegExps from shared class strings instead of upstream's inline literals, so each table ships in the bundle once (roughly 6 KB of Unicode data instead of 18).Two deletion behaviors follow from the new boundary. Forward delete at the start of a cluster now removes the whole cluster instead of only the leading consonant and virama; a browser test pins it and fails without the fix. Backspace keeps removing one code unit per press: the whole-cluster delete composes with the compound-script reinsert in
delete-internal.ts. That reinsert was keyed to script ranges that miss Gujarati, Tai Tham, Balinese, Sundanese, Javanese, and Meetei Mayek, all of which GB9c now glues, so the regex additionally matches those scripts' BMP conjunct linkers (every GB9c cluster contains its linker). Browser tests pin the backspace parity for Devanagari (in-range script) and Gujarati (linker-keyed, red without the regex extension). The supplementary-plane linkers (Kharoshthi, Chakma, Kawi, and kin) are left out deliberately, since the reinsert slices code units and would split a surrogate pair; backspace on those scripts removes the whole cluster, and nothing pins them.The grapheme walker itself is pinned by unit tests adapted from upstream's samples, asserted in both LTR and RTL directions, red without the fix. A small test-only commit rides along: it pins that backspacing a ZWJ emoji sequence removes the whole cluster (emoji are outside the compound-script reinsert), pre-existing behavior that nothing guarded.
Note
Medium Risk
Changes grapheme walking and character-delete for Indic scripts, so caret and Backspace/Delete behavior can regress if GB9c tables or the compound-script reinsert are wrong. Not auth or data-handling, but it is core text-editing.
Overview
Caret movement and character positions now treat Indic conjunct clusters (consonant + virama + consonant, e.g. Devanagari क्ष) as one grapheme, matching browser rendering and Unicode UAX #29 GB9c.
getCharacterDistancegains InCB Consonant/Extend/Linker classes, a GB9c non-boundary pair, and a deferred left-context check so the caret no longer lands inside a cluster. Forward Delete at the start of a cluster removes the whole cluster. Backspace still peels one code unit at a time via the compound-script reinsert inapplyDelete; that regex now also matches BMP conjunct linkers for scripts the old ranges missed (e.g. Gujarati). Supplementary-plane linkers are left out so reinsert does not split surrogates.Unit tests cover GB9c in LTR/RTL; browser tests pin Indic/Gujarati backspace, forward delete of a conjunct, and whole-cluster backspace of ZWJ emoji.
Reviewed by Cursor Bugbot for commit dac0376. Bugbot is set up for automated code reviews on this repo. Configure here.