[fix](streaming-job) Encode CDC stream load records as UTF-8

[JNSimba]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

CDC streaming jobs encoded JSON records with String.getBytes(), so the stream-load payload depended on the CDC Client JVM default charset. On JVMs using a non-UTF-8 default charset, characters such as Chinese text and emoji were replaced with question marks before reaching Doris. Encode records explicitly as UTF-8, matching the existing CDC fetch path and the stream-load JSON contract.

Before the change, MySqlCharsetITCase running with -Dfile.encoding=US-ASCII failed with expected: "测试数据😀" but was: "?????". After the change, the same test passes without requiring the CDC Client JVM default charset to be UTF-8.

Release note

Fix CDC streaming jobs corrupting non-ASCII text when the CDC Client JVM uses a non-UTF-8 default charset.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

  mvn -Dtest=MySqlCharsetITCase "-Dtest.add.opens=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED -Dfile.encoding=US-ASCII" test

• Behavior changed:

  • No.
  • Yes. CDC stream-load records are always encoded as UTF-8.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[JNSimba]

run buildall

[JNSimba]

/review

[github-actions]

Review status: completed after two rounds; one blocking correctness issue is reported inline.

Checkpoint conclusions:

• Goal, data correctness, and tests: Explicit UTF-8 encoding at the changed call correctly preserves each deserialized JSON record, and the documented US-ASCII MySqlCharsetITCase path reaches both snapshot and binlog conversion. The complete multi-row stream-load entity is still default-charset-dependent because its separator is encoded elsewhere with the JVM default; the current one-row-per-flush charset test misses that boundary, so the stated goal is not fully achieved.
• Scope and parallel paths: This is a small, clear record-level change. writeRecords is the sole production DML writeRecord caller, while the direct fetch path already uses UTF-8 for both records and delimiters. No second distinct record path issue was found.
• Concurrency and lifecycle: The conversion is local and hands the existing sink a fresh byte array. It adds no shared state, lock interaction, thread, cleanup obligation, static initialization dependency, or ownership change.
• Configuration, compatibility, and persistence: No new configuration, FE/BE field, persisted state, storage format, or rolling-upgrade protocol is introduced. UTF-8-default deployments are byte-identical, but cdc_client_java_opts and adopted external clients make non-ASCII-compatible defaults reachable. Existing stream-load success/error handling and offset-commit ordering are otherwise unchanged.
• Performance and observability: The old path already allocated one byte array per record; selecting a constant charset adds no material copy, scan, lock, metric, or logging requirement.
• Test execution: The review environment prohibits builds/tests. The current CDC build/test checks are green, but their standard charset run does not cover the accepted multi-record UTF-16 framing case.
• User focus: No additional user-provided focus was supplied; the full PR was reviewed.

All candidates were independently verified and deduplicated. Round 2 converged with no new valuable findings beyond the inline issue.

[Copilot]

Pull request overview

This PR fixes character corruption in CDC streaming jobs by ensuring JSON stream-load records are always encoded as UTF-8, independent of the CDC client JVM’s default charset. This aligns the stream-load payload encoding with the JSON contract and avoids data loss for non-ASCII text.

Changes:

• Encode stream-load record payloads using StandardCharsets.UTF_8 instead of relying on String.getBytes() default charset behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.