You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "FayneBupt (via GitHub)" <gi...@apache.org> on 2026/03/31 12:44:19 UTC
[PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
FayneBupt opened a new pull request, #61963:
URL: https://github.com/apache/doris/pull/61963
### What problem does this PR solve?
Issue Number: close #56005
Related PR: #xxx
Problem Summary:
This PR implements Doris write-back support for Paimon external tables (`INSERT INTO ...`), completing the end-to-end write path across FE and BE, including planning, sink serialization, execution, and FE-side commit.
#### 1) Core capability
- Support writing data from Doris into Paimon external tables.
- Support both writer paths:
- JNI writer path
- paimon-cpp/native writer path
#### 2) FE/BE write pipeline integration
- FE:
- Add Paimon insert executor flow.
- Build and send Paimon sink metadata (table identity/location/options/column names/shuffle info).
- Collect commit payload from BE and commit on FE side.
- BE:
- Execute Paimon sink writing.
- Build commit payload and report back to FE.
#### 3) Bucket/partition write behavior
- Add distributed bucket shuffle support for bucketed Paimon tables.
- Add compatible fallback behavior (e.g. gather/random) when required bucket keys are not resolvable.
- Keep partition-related write behavior aligned with existing sink framework.
#### 4) Session variables and write tunables
This PR introduces/uses Paimon-related session controls for write behavior and performance tuning, including:
- `enable_paimon_jni_writer`
- `enable_paimon_distributed_bucket_shuffle`
- `enable_paimon_adaptive_buffer_size`
- `paimon_writer_queue_size`
- `paimon_target_file_size`
- `paimon_write_buffer_size`
- `enable_paimon_jni_spill`
- `paimon_spill_max_disk_size`
- `paimon_spill_sort_buffer_size`
- `paimon_spill_sort_threshold`
- `paimon_spill_compression`
- `paimon_global_memory_pool_size`
#### 5) Compatibility/alignment work
- Align merged code with current community branch conventions and dependencies.
- Resolve compile/checkstyle conflicts introduced by branch differences.
- Ensure FE build passes under Maven >= 3.9 environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4214668029
# BE Regression && UT Coverage Report
Increment line coverage `2.52% (45/1783)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61963_062570f2d2192061a2b8626395f0378b4a79c599_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61963_062570f2d2192061a2b8626395f0378b4a79c599_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.61% (26726/37320) |
| Line Coverage | 54.67% (283127/517889) |
| Region Coverage | 51.80% (234857/453398) |
| Branch Coverage | 53.21% (101490/190749) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4182008444
# BE Regression && UT Coverage Report
Increment line coverage `87.34% (400/458)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61963_0d8409fea67b05c71d100d5b9940330a882d6dd7_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61963_0d8409fea67b05c71d100d5b9940330a882d6dd7_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.60% (27311/37107) |
| Line Coverage | 57.20% (294007/513954) |
| Region Coverage | 54.35% (244580/450016) |
| Branch Coverage | 56.10% (106141/189205) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4206376343
# FE UT Coverage Report
Increment line coverage ` 16.72% (96/574)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61963_185a76210c364194c9365f72ce48d8761108f483/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61963_185a76210c364194c9365f72ce48d8761108f483/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4170601141
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ea3631a7cc0a5d3e1b798af2cd52cf2fc385aadb_ea3631a7cc0a5d3e1b798af2cd52cf2fc385aadb_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ea3631a7cc0a5d3e1b798af2cd52cf2fc385aadb_ea3631a7cc0a5d3e1b798af2cd52cf2fc385aadb_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.48% (1798/2291) |
| Line Coverage | 64.14% (32269/50310) |
| Region Coverage | 64.99% (16174/24886) |
| Branch Coverage | 55.49% (8625/15542) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4213694302
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4253143818
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4230012827
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4181656411
<details>
<summary>TPC-DS: <b>Total hot run time: 179865 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d8409fea67b05c71d100d5b9940330a882d6dd7, data reload: false
query5 4367 662 505 505
query6 333 235 221 221
query7 4245 555 362 362
query8 337 261 237 237
query9 8738 3924 3957 3924
query10 507 398 346 346
query11 6653 5509 5142 5142
query12 203 137 132 132
query13 1301 624 460 460
query14 5808 5240 4826 4826
query14_1 4196 4186 4105 4105
query15 216 205 185 185
query16 1027 461 459 459
query17 1163 804 689 689
query18 2783 498 384 384
query19 251 227 203 203
query20 144 130 130 130
query21 221 150 121 121
query22 13642 13538 13401 13401
query23 17929 17723 16731 16731
query23_1 16969 16770 16722 16722
query24 8106 1844 1480 1480
query24_1 1446 1446 1480 1446
query25 649 511 505 505
query26 1312 362 190 190
query27 2936 645 359 359
query28 4478 1943 1919 1919
query29 973 671 550 550
query30 304 232 193 193
query31 1098 1059 948 948
query32 87 71 71 71
query33 543 353 288 288
query34 1179 1203 682 682
query35 753 773 661 661
query36 1226 1207 1044 1044
query37 154 101 85 85
query38 3119 3065 2969 2969
query39 914 890 855 855
query39_1 835 833 847 833
query40 244 165 142 142
query41 64 58 58 58
query42 273 275 276 275
query43 315 328 284 284
query44
query45 208 201 193 193
query46 1141 1282 804 804
query47 2316 2356 2221 2221
query48 402 440 334 334
query49 654 526 461 461
query50 699 280 209 209
query51 4372 4268 4232 4232
query52 282 284 269 269
query53 338 362 287 287
query54 331 294 288 288
query55 102 95 94 94
query56 336 331 318 318
query57 1747 1702 1598 1598
query58 300 278 282 278
query59 2909 3010 2713 2713
query60 332 343 335 335
query61 161 150 159 150
query62 674 624 571 571
query63 325 304 300 300
query64 5310 1440 1100 1100
query65
query66 1439 478 382 382
query67 24276 24265 24183 24183
query68
query69 458 347 320 320
query70 1046 1004 973 973
query71 368 351 333 333
query72 3040 2920 2655 2655
query73 815 762 457 457
query74 9885 9750 9554 9554
query75 3576 3392 3021 3021
query76 2326 1151 781 781
query77 407 440 356 356
query78 11424 11431 10746 10746
query79 1494 1106 824 824
query80 1362 768 681 681
query81 522 277 240 240
query82 1023 165 125 125
query83 373 290 262 262
query84 267 147 121 121
query85 928 509 476 476
query86 466 333 317 317
query87 3292 3215 3104 3104
query88 3564 2687 2672 2672
query89 503 415 390 390
query90 1979 182 179 179
query91 177 177 145 145
query92 79 77 68 68
query93 1029 902 511 511
query94 710 339 316 316
query95 648 458 340 340
query96 997 773 332 332
query97 2668 2654 2567 2567
query98 247 230 232 230
query99 1083 1061 971 971
Total cold run time: 260643 ms
Total hot run time: 179865 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4181623997
<details>
<summary>TPC-H: <b>Total hot run time: 29567 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d8409fea67b05c71d100d5b9940330a882d6dd7, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17634 3720 3709 3709
q2 q3 10679 877 610 610
q4 4681 463 363 363
q5 7485 1345 1151 1151
q6 188 172 141 141
q7 923 957 758 758
q8 9308 1420 1334 1334
q9 5563 5302 5285 5285
q10 6272 2042 1800 1800
q11 496 285 287 285
q12 822 698 515 515
q13 18064 2794 2175 2175
q14 287 281 261 261
q15 q16 864 873 791 791
q17 1100 1054 848 848
q18 6487 5707 5565 5565
q19 1176 1292 1087 1087
q20 573 543 418 418
q21 4300 2501 2089 2089
q22 503 434 382 382
Total cold run time: 97405 ms
Total hot run time: 29567 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4674 4484 4634 4484
q2 q3 4612 4712 4198 4198
q4 2023 2191 1395 1395
q5 4846 5003 5263 5003
q6 194 171 135 135
q7 2054 1791 1646 1646
q8 3335 3070 3087 3070
q9 8266 8325 8253 8253
q10 4432 4434 4205 4205
q11 588 434 401 401
q12 665 723 495 495
q13 2777 3048 2363 2363
q14 292 293 268 268
q15 q16 768 791 682 682
q17 1278 1371 1233 1233
q18 7737 6974 7061 6974
q19 1119 1143 1133 1133
q20 2229 2235 1940 1940
q21 6007 5350 4744 4744
q22 550 513 430 430
Total cold run time: 58446 ms
Total hot run time: 53052 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61963:
URL: https://github.com/apache/doris/pull/61963#discussion_r3068448141
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/LoadProcessor.java:
##########
@@ -187,13 +188,19 @@ protected void doProcessReportExecStatus(TReportExecStatusParams params, SingleF
}
}
+ LoadContext loadContext = coordinatorContext.asLoadProcessor().loadContext;
+ long txnId = loadContext.getTransactionId();
Review Comment:
`txnId` is read from `loadContext` before this method applies `params.getTxnId()` to the context. On the first terminal report from a fragment, `loadContext.getTransactionId()` is still unset, so `paimonCommitMessages` are dropped here and never reach `PaimonTransaction`. `PaimonTransaction.commit()` then sees an empty list and returns without publishing anything. Please derive the transaction id from `params` first (or move this block after the `params.isSetTxnId()` update) so the first commit payload report is not lost.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/PaimonTableSink.java:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.datasource.mvcc.MvccUtil;
+import org.apache.doris.datasource.paimon.PaimonExternalCatalog;
+import org.apache.doris.datasource.paimon.PaimonExternalTable;
+import org.apache.doris.nereids.trees.plans.commands.insert.BaseExternalTableInsertCommandContext;
+import org.apache.doris.nereids.trees.plans.commands.insert.InsertCommandContext;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.thrift.TDataSink;
+import org.apache.doris.thrift.TDataSinkType;
+import org.apache.doris.thrift.TExplainLevel;
+import org.apache.doris.thrift.TFileFormatType;
+import org.apache.doris.thrift.TPaimonTableSink;
+import org.apache.doris.thrift.TPaimonWriteShuffleMode;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.utils.InstantiationUtil;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Paimon table sink
+ *
+ * This class materializes the TPaimonTableSink payload consumed by BE,
+ * including table location, write options, partition keys, bucket keys,
+ * shuffle mode and sink column names.
+ */
+public class PaimonTableSink extends BaseExternalTableDataSink {
+ private static final Logger LOG = LogManager.getLogger(PaimonTableSink.class);
+ private final PaimonExternalTable targetTable;
+ private static final Base64.Encoder BASE64_ENCODER = java.util.Base64.getUrlEncoder().withoutPadding();
+ private static final HashSet<TFileFormatType> supportedTypes = new HashSet<TFileFormatType>() {{
+ add(TFileFormatType.FORMAT_ORC);
+ add(TFileFormatType.FORMAT_PARQUET);
+ }};
+
+ public PaimonTableSink(PaimonExternalTable targetTable) {
+ super();
+ this.targetTable = targetTable;
+ }
+
+ // List of columns to be written to the sink, used to populate columnNames in Thrift
+ private List<Column> cols;
+
+ public void setCols(List<Column> cols) {
+ this.cols = cols;
+ }
+
+ @Override
+ public String getExplainString(String prefix, TExplainLevel explainLevel) {
+ StringBuilder strBuilder = new StringBuilder();
+ strBuilder.append(prefix).append("PAIMON TABLE SINK\n");
+ if (explainLevel == TExplainLevel.BRIEF) {
+ return strBuilder.toString();
+ }
+ return strBuilder.toString();
+ }
+
+ @Override
+ protected Set<TFileFormatType> supportedFileFormatTypes() {
+ return supportedTypes;
+ }
+
+ @Override
+ public void bindDataSink(Optional<InsertCommandContext> insertCtx) throws AnalysisException {
+ TPaimonTableSink tSink = new TPaimonTableSink();
+
+ // basic identifiers
+ tSink.setCatalogName(targetTable.getCatalog().getName());
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTbName(targetTable.getName());
+
+ Map<String, String> catalogProps = targetTable.getCatalog().getCatalogProperty().getHadoopProperties();
+ Map<String, String> options = new HashMap<>();
+ options.putAll(catalogProps);
+ String warehouse = ((PaimonExternalCatalog) targetTable.getCatalog()).getPaimonOptionsMap()
+ .get(CatalogOptions.WAREHOUSE.key());
+ String defaultFsName = resolveDefaultFsName(warehouse);
+ if (defaultFsName != null && !defaultFsName.isEmpty()) {
+ String currentDefaultFs = options.get("fs.defaultFS");
+ if (currentDefaultFs == null || currentDefaultFs.isEmpty() || currentDefaultFs.startsWith("file:/")) {
+ options.put("fs.defaultFS", defaultFsName);
+ }
+ }
+ if (insertCtx.isPresent() && insertCtx.get() instanceof BaseExternalTableInsertCommandContext) {
+ BaseExternalTableInsertCommandContext ctx = (BaseExternalTableInsertCommandContext) insertCtx.get();
+ if (ctx.getTxnId() > 0) {
+ options.put("doris.commit_identifier", String.valueOf(ctx.getTxnId()));
+ }
+ if (ctx.getCommitUser() != null && !ctx.getCommitUser().isEmpty()) {
+ options.put("doris.commit_user", ctx.getCommitUser());
+ }
+ }
+
+ if (ConnectContext.get() != null) {
+ options.put("target-file-size",
+ String.valueOf(ConnectContext.get().getSessionVariable().paimonTargetFileSize));
+ options.put("write-buffer-size",
+ String.valueOf(ConnectContext.get().getSessionVariable().paimonWriteBufferSize));
+ String hadoopUser = options.get("hadoop.username");
+ if (hadoopUser == null || hadoopUser.isEmpty()) {
+ hadoopUser = options.get("hadoop.user.name");
+ }
+ if (hadoopUser == null || hadoopUser.isEmpty()) {
+ hadoopUser = "hadoop";
+ }
+ options.put("hadoop.user.name", hadoopUser);
+ options.put("hadoop.username", hadoopUser);
+ }
+
+ String tableLocation = null;
+ org.apache.paimon.table.Table paimonTable =
+ targetTable.getPaimonTable(MvccUtil.getSnapshotFromContext(targetTable));
+ if (paimonTable instanceof org.apache.paimon.table.FileStoreTable) {
+ tableLocation = ((org.apache.paimon.table.FileStoreTable) paimonTable).location().toString();
+ }
+ tableLocation = normalizeTableLocation(tableLocation);
+ if (tableLocation == null || tableLocation.isEmpty()) {
+ if (warehouse != null && !warehouse.isEmpty()) {
+ String base = warehouse.endsWith("/") ? warehouse : warehouse + "/";
+ tableLocation = base + targetTable.getDbName() + ".db/" + targetTable.getName() + "/";
+ }
+ } else if (defaultFsName != null && !defaultFsName.isEmpty() && tableLocation.startsWith("hdfs://")) {
Review Comment:
This rewrites every explicit `hdfs://<authority>/...` table location to `defaultFsName + path`, which discards the authority returned by Paimon metadata. If the warehouse authority differs from the table authority, FE will send BE to write to the wrong filesystem location. `FileStoreTable.location()` is already authoritative here; only the authority-missing case should be normalized, not the fully-qualified `hdfs://...` case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4206157527
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/185a76210c364194c9365f72ce48d8761108f483_185a76210c364194c9365f72ce48d8761108f483_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/185a76210c364194c9365f72ce48d8761108f483_185a76210c364194c9365f72ce48d8761108f483_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.48% (1798/2291) |
| Line Coverage | 64.12% (32259/50312) |
| Region Coverage | 65.05% (16232/24954) |
| Branch Coverage | 55.61% (8680/15610) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-6135217193
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4211226821
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4205355766
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4181991251
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0d8409fea67b05c71d100d5b9940330a882d6dd7_0d8409fea67b05c71d100d5b9940330a882d6dd7/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0d8409fea67b05c71d100d5b9940330a882d6dd7_0d8409fea67b05c71d100d5b9940330a882d6dd7/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.94% (20057/37887) |
| Line Coverage | 36.55% (188417/515524) |
| Region Coverage | 32.84% (146431/445877) |
| Branch Coverage | 33.97% (64067/188623) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #61963: [feat](paimon) support paimon insert by jni&paimon-cpp
URL: https://github.com/apache/doris/pull/61963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4169842096
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4212331579
# BE UT Coverage Report
Increment line coverage `1.18% (21/1783)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3afcf1a553f1e7650591e948aa6a42654ae1cbf7_3afcf1a553f1e7650591e948aa6a42654ae1cbf7/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3afcf1a553f1e7650591e948aa6a42654ae1cbf7_3afcf1a553f1e7650591e948aa6a42654ae1cbf7/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.82% (20125/38104) |
| Line Coverage | 36.42% (189181/519481) |
| Region Coverage | 32.69% (146899/449303) |
| Branch Coverage | 33.79% (64258/190188) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4205904743
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4230251034
### High Priority Issues
#### 1. Missing end-to-end regression tests (Blocker)
There are no integration tests under `regression-test/`. The existing FE unit tests (`PaimonTransactionTest`, `PaimonWriteBufferSizeTest`) only cover serialization and buffer size helpers. We need at minimum:
- Non-partitioned table INSERT INTO + read-back verification
- Partitioned table INSERT INTO
- Fixed-bucket table INSERT INTO
- Type coverage (STRING, INT, DECIMAL, TIMESTAMP, ARRAY, MAP, STRUCT)
- Error scenarios (non-existent table, schema mismatch)
Also missing: UT for `BindSink.bindPaimonTableSink()`, `PaimonTableSink.bindDataSink()`, `PhysicalPaimonTableSink.getRequirePhysicalProperties()`, and BE-side `VPaimonTableWriter::write()`.
#### 2. `_extract_write_key()` per-row performance bottleneck
`_extract_write_key()` calls `type->to_string(*col, row, options)` for **every row** in the write path. For multi-million row writes, the repeated string conversion and map lookups introduce non-trivial overhead. The Iceberg Writer uses column-level batch partition calculation.
**Suggestion**: Compute partition keys at the block level — extract unique partition value sets per block, then batch-dispatch rows by group.
#### 3. `VPaimonTableWriter` copies the entire `TDataSink` by value
`_t_sink` is declared as `const TDataSink` (value type). Since `TDataSink` contains `serialized_table` (a full Base64-encoded Paimon Table object), this copy can be tens of KB to several MB.
```diff
// be/src/vec/sink/vpaimon_table_writer.h
- const TDataSink _t_sink;
+ const TDataSink& _t_sink;
```
Note: If switching to a reference, the `TDataSink` owner's lifetime must be guaranteed to outlive the writer. Alternatively, use `std::shared_ptr<const TDataSink>`.
#### 4. `VPaimonTableWriter::close()` — `Defer` timing issue with commit messages
```cpp
Defer do_close {[&]() { close_status = _file_store_write->Close(); }};
// PrepareCommit() runs here, adds messages to _state
// ... then do_close fires on scope exit
```
If `PrepareCommit()` succeeds and commit messages are added to `_state`, but the subsequent `Close()` fails (inside `Defer`), the method returns an error status even though commit messages have already been submitted. This creates an inconsistent state — FE sees commit messages but BE reports failure.
**Suggestion**: Move `Close()` out of `Defer` and call it explicitly after `PrepareCommit`, or clear committed messages if `Close()` fails.
---
### Medium Priority Issues
#### 5. `deserializeCommitMessagePayload()` log output can be unbounded
When the DPCM header fails to parse, this line dumps the entire payload byte array at WARN level:
```java
LOG.warn("paimon: payload header mismatch or too short: length={}, payload={}",
payload == null ? 0 : payload.length, java.util.Arrays.toString(payload));
```
In production, payloads can be megabytes. The brute-force fallback (trying versions 11→3, up to 9 exception catches) also has performance implications.
```diff
- LOG.warn("paimon: payload header mismatch or too short: length={}, payload={}",
- payload == null ? 0 : payload.length, java.util.Arrays.toString(payload));
+ LOG.warn("paimon: payload header mismatch or too short: length={}",
+ payload == null ? 0 : payload.length);
+ LOG.debug("paimon: problematic payload bytes: {}",
+ payload == null ? "null" : java.util.Arrays.toString(payload));
```
#### 6. `PaimonDorisMemoryPool::_ensure_thread_context()` — thread_local lifetime concern
`tls_attach` is `static thread_local` and never released until the thread is destroyed. In thread-pool reuse scenarios, the previously attached query tracker may have expired. Consider checking tracker validity on each invocation or using scoped management.
#### 7. INSERT OVERWRITE may be incomplete
`UnboundTableSinkCreator.createUnboundTableSinkMaybeOverwrite()` adds a Paimon branch but `UnboundPaimonTableSink` does not propagate `staticPartitionKeyValues` or `isAutoDetectPartition`, unlike the Iceberg implementation. The INSERT OVERWRITE semantics may be incomplete.
---
### Low Priority Issues
#### 8. `PaimonTableCache` constants should be configurable
`maxExternalSchemaCacheNum = 50` and `expireAfterAccess = 100 minutes` are hardcoded. For environments managing hundreds of Paimon tables with periodic writes, 50 cache entries may be insufficient.
#### 9. Dynamic bucket retry in `PaimonJniWriter.writeBatch()` is fragile
```java
if (t instanceof IllegalArgumentException
&& t.getMessage() != null
&& t.getMessage().contains("dynamic bucket mode")) {
writer.write(reusedRow, 0); // retry with bucket=0
}
```
Error handling based on exception message string matching is brittle. Consider detecting dynamic bucket mode during `open()` and storing it as a flag, rather than relying on runtime exception matching.
#### 10. Logging level issues
| File | Issue |
|---|---|
| `PaimonTableCache.java:47` | `LOG.warn("load table:{}", key)` — Cache miss is not a warning condition; should be `LOG.info` |
| `PaimonInsertExecutor.java` | No log statements in `beforeExec()` or `doBeforeCommit()`; consider adding key operation logging |
#### 11. `rawlog_compat.cpp` and `rawlog_compat_paimon.cpp` have duplicated formatting logic
Both files implement nearly identical log formatting. Consider extracting a shared internal function.
The remaining items are medium/low priority and can be addressed in follow-up iterations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4168718146
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4162383457
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "FayneBupt (via GitHub)" <gi...@apache.org>.
FayneBupt commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4181398035
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](paimon) support paimon insert by jni&paimon-cpp [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61963:
URL: https://github.com/apache/doris/pull/61963#issuecomment-4181541350
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/0d8409fea67b05c71d100d5b9940330a882d6dd7_0d8409fea67b05c71d100d5b9940330a882d6dd7_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/0d8409fea67b05c71d100d5b9940330a882d6dd7_0d8409fea67b05c71d100d5b9940330a882d6dd7_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.48% (1798/2291) |
| Line Coverage | 64.16% (32278/50310) |
| Region Coverage | 65.00% (16177/24886) |
| Branch Coverage | 55.50% (8626/15542) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org