You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "lszskye (via GitHub)" <gi...@apache.org> on 2026/09/15 10:34:09 UTC
[PR] perf(write): write input changelog and data files in parallel [paimon-cpp]
lszskye opened a new pull request, #350:
URL: https://github.com/apache/paimon-cpp/pull/350
### Purpose
Linked issue: #174
Improve INPUT changelog write performance by writing data and changelog
files concurrently.
This change:
- writes changelog files asynchronously while data files are written on
the caller thread;
- waits for both write pipelines before publishing their results;
- aborts both writers when either pipeline fails;
- adds `changelog-producer.input.parallel-write`, enabled by default;
- preserves the previous sequential behavior when the option is disabled;
- extracts common sort-merge writing logic;
- reuses a shared helper when creating data and changelog rolling writers.
### Tests
- Extended `CoreOptionsTest` to verify the default and configured values
of the new option.
- Updated `MergeTreeWriterTest.TestInputChangelog` to support
nondeterministic file numbering during parallel writes.
- Added `MergeTreeWriterTest.TestInputChangelogWithParallelWriteDisabled`
to verify the sequential fallback and file creation order.
### API and Format
This change adds the public option
`changelog-producer.input.parallel-write`.
It does not change the storage format or protocol.
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(write): write input changelog and data files in parallel [paimon-cpp]
Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 commented on code in PR #350:
URL: https://github.com/apache/paimon-cpp/pull/350#discussion_r4023720501
##########
include/paimon/defs.h:
##########
@@ -421,6 +421,10 @@ struct PAIMON_EXPORT Options {
/// "changelog-producer.row-deduplicate" is "true".
static const char CHANGELOG_PRODUCER_ROW_DEDUPLICATE_IGNORE_FIELDS[];
+ /// "changelog-producer.input.parallel-write" - Whether data and changelog files are
+ /// written in parallel for input mode changelog producer. Default value is "true".
+ static const char CHANGELOG_PRODUCER_INPUT_PARALLEL_WRITE[];
+
Review Comment:
Please clarify that this is specific to C++.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(write): write input changelog and data files in parallel [paimon-cpp]
Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 merged PR #350:
URL: https://github.com/apache/paimon-cpp/pull/350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org