Skip to content

Fix character encoding problem for non-ASCII/Chinese data (#795) - #1693

Open
sanjaykumarmtt wants to merge 6 commits into
eclipse-vertx:masterfrom
sanjaykumarmtt:fix-db2-encoding-795
Open

Fix character encoding problem for non-ASCII/Chinese data (#795)#1693
sanjaykumarmtt wants to merge 6 commits into
eclipse-vertx:masterfrom
sanjaykumarmtt:fix-db2-encoding-795

Conversation

@sanjaykumarmtt

Copy link
Copy Markdown

### Motivation
Fixes #795

When reading Chinese or other multi byte characters from DB2 they show up as ??? because the driver only checked OS type instead of the server CCSID.

This change fixes the issue:

  • Maps DB2 CCSID numbers to Java Charset in CCSIDConstants.
  • Uses server column CCSID (sda.ccsid_) if present; otherwise, uses connection level encoding (getCcsidMbcEncoding()).

Conformance

  • I have signed the Eclipse Contributor Agreement (ECA).
  • Followed code style and tested with DB2 database.

@tsegismont tsegismont left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @sanjaykumarmtt

Could you please add a test?

@sanjaykumarmtt

Copy link
Copy Markdown
Author

@tsegismont thank you for the review I have added the unit test and formatted the code accordingly

@tsegismont tsegismont left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding a test, but I was thinking about actually testing a DB2 query rather than a unit test. Is it possible?

@sanjaykumarmtt

Copy link
Copy Markdown
Author

thanks for the feedback @tsegismont yes that makes sense I will add a DB2 query integration test for this and update the PR soon

@sanjaykumarmtt

Copy link
Copy Markdown
Author

hi @tsegismont ​apologies for the delay I have replaced the previous unit test with a DB2 query integration test in DB2QueriesTest as suggested please review it and let me know if any further changes are needed

tsegismont and others added 2 commits September 7, 2026 17:03
@sanjaykumarmtt

Copy link
Copy Markdown
Author

​I have updated the PR with the suggested fix: casting the untyped parameter marker with CAST(? AS VARCHAR(100)) in DB2QueriesTest, and reverted the unrelated changes.
​Could you please approve and trigger the CI workflow when you have a moment?

@tsegismont tsegismont left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. For me the test passes without the code changes. Can you check if it's actually testing what you want?

@sanjaykumarmtt

Copy link
Copy Markdown
Author

sorry for the delay ​I found why the test passed earlier the target chinese string was not present in the database table so the query returned 0 rows and did not properly test the encoding.

@sanjaykumarmtt

Copy link
Copy Markdown
Author

​I have updated the test ​added the Chinese string to init.sql ​updated the test assertion to verify that 1 row is returned and the decoded message matches the expected value.

@sanjaykumarmtt

Copy link
Copy Markdown
Author

​Could you please approve and trigger the CI workflow when you have a moment?

@tsegismont

Copy link
Copy Markdown
Member

The test still passes without the changes. I think this happens because it's all UTF-8.

Please make sure to create a test that does not pass without the changes.

@sanjaykumarmtt

Copy link
Copy Markdown
Author

hi @tsegismont the original test was passing without changes because the database environment and connection default to UTF-8 CCSID 1208 which bypasses the specific Chinese CCSID mapping logic

to trigger the mapping I tried casting the column using CAST('你好世界' AS VARCHAR(100) CCSID 1386) but DB2 threw SQLCODE=-622 (SQLSTATE=56031) because this database configuration does not support that clause

@sanjaykumarmtt

Copy link
Copy Markdown
Author

I tried writing a dedicated unit test (CCSIDConstantsTest) locally to verify the mapping and it passes with the changes and fails without them
Could you please guide me on whether I should commit this unit test or if you have any suggestions on how to test this properly?

@tsegismont

Copy link
Copy Markdown
Member

I'm not a DB2 expert unfortunately. Perhaps is it possible to create a specific database with different encoding?

@sanjaykumarmtt

Copy link
Copy Markdown
Author

​to be honest I m not a DB2 expert at all I recently graduated from college where I built a chat app using Java sockets and I really wanted to contribute to a real world open source project like vert.x. earlier I only tried casting the encoding inline at the query level to see if it would trigger the mapping code.

​after your suggestion I read up on creating a dedicated non UTF 8 database like GBK or CCSID 1386 for testing but I noticed a few practical drawbacks:
​Slower CI builds: Running CREATE DATABASE ... USING CODESET GBK in DB2 takes an extra 3 to 5 minutes on every CI run.
​CI instability: DB2 is resource-heavy, and creating an additional database during testing can easily lead to timeouts or memory crashes in GitHub Actions runners.
​Scope of the fix: Our fix is purely Java-side logic in CCSIDConstants to map CCSID numbers to standard Java Charsets.

because of this the dedicated unit test CCSIDConstantsTest feels like the cleanest and safest solution It reliably asserts that CCSIDs 1386, 5488, 1114 map to GBK and 935 maps to Cp935 It fails on master and passes cleanly with this fix
​would it be okay if I push this unit test commit to the PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vertx-db2-client character encoding problem

2 participants