Fix/use pure connector 8kb truncation - #13
Open
ducanhtran1796 wants to merge 15 commits into
Open
ducanhtran1796 wants to merge 15 commits into
ducanhtran1796 wants to merge 15 commits into
Conversation
refactor handle_schema_evolution
refactor handle_schema_evolution
mysql-connector ships a C extension that is used by default whenever it is
installed (`use_pure` defaults to False in mysql.connector.connect). That
extension pushes statements through a fixed NET_BUFFER_LENGTH (8192) buffer,
and Doris' FE does not reassemble the resulting split packets: anything longer
than 8KB reaches the parser truncated mid-token.
The failure is confusing because it surfaces as a syntax error pointing at
whichever line happens to contain byte 8192, with a garbled token stream:
no viable alternative at input 'CAST(JSONJSON(JSON'(line 155, pos 20)
A model that flattens ~100 JSON columns crosses 8KB easily, so this is hit by
any reasonably wide model rather than being an edge case.
Add `use_pure` to DorisCredentials, defaulting to True, and pass it through to
mysql.connector.connect(). Set `use_pure: false` in profiles.yml to opt back
into the C extension if every statement stays under 8KB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.