You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Balazs Hevele (Code Review)" <ge...@cloudera.org> on 2026/04/14 12:59:28 UTC
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Balazs Hevele has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24190
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/paimon/paimon-jni-row-reader.cc
M be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
5 files changed, 124 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/1
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 1
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 24:
(5 comments)
High level comments, didn't look deeply into tests/benchmarks/paimon code.
For optimization ideas feel free to postpone them to a later commit.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@84
PS24, Line 84: if (arrow_array_->IsNull(row_idx)) {
A potential optimization could be to optimize for cases with no NULLs to avoid this check per row.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@89
PS24, Line 89: tuple->SetNotNull(null_indicator_offset);
It may be faster to set all tuples to 0 (meaning non-null) and only set null bit in the NULL cases.
Impala has template_tuple_ for this - set all tuples do some basic value (e.g. set partition columns), and the set remaining slots individually.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@138
PS24, Line 138: template<class T, class AT>
Can you add a comment about these types? Or a more descriptive name could help. I assume that it is Type and ArrowType, but needed some time.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@214
PS24, Line 214: WriteSlotInternal
Some of this code could be possibly merged with Parquet scanning code.
ParquetPlainEncoder::Decode does similar work as some of these functions, so copies data with Parquet type to an Impala slot. Generally Iceberg types match Parquet plain encoded types, though null handling is different.
For timestamps, see ParquetTimestampDecoder's Int64ToTimestampValue.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@377
PS24, Line 377: local_time_zone
Does arrow store timestamps as UTC? Iceberg uses mainly timestamps without time zone, which means that we assume that no timezone conversion is needed.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 24
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Jul 2026 10:15:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#24).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
12 files changed, 1,233 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/24
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 24
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#8).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
9 files changed, 516 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/8
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 8
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22192/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 7
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Apr 2026 13:10:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 10:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22237/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 10
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 25 Apr 2026 18:46:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#37).
Change subject: WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
13 files changed, 1,384 insertions(+), 520 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/37
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 37
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 28:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23406/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 28
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 03:54:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22300/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 May 2026 09:00:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#17).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 1,167 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/17
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 17
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22131/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 1
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Apr 2026 13:22:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 27:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23405/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 27
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 03:13:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#36).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
13 files changed, 1,384 insertions(+), 520 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/36
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 36
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#33).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
12 files changed, 1,308 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/33
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 33
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#5).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/paimon/paimon-jni-row-reader.cc
M be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
5 files changed, 241 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/5
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 5
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#4).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/paimon/paimon-jni-row-reader.cc
M be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
5 files changed, 135 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/4
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 4
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#9).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
9 files changed, 546 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/9
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 9
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#7).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/paimon/paimon-jni-row-reader.cc
M be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
5 files changed, 360 insertions(+), 235 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/7
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 7
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 32:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23859/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 32
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 09:52:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 35:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc
File be/src/benchmarks/arrow-converter-benchmark.cc:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc@48
PS33, Line 48: // Mixed 271 277 281 0.238X 0.237X 0.234X
> Added an arrow to arrow copy suite for comparison.
Done
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/arrow-converter.cc@54
PS33, Line 54: RETURN_IF_ERROR(WriteSlotBatch(arr.get(), 0, num_rows, slot_desc,
> I don't understand what you mean exactly.
This essentially iterates row by row (tuple_mem represents a single tuple). Our memory layout doesn't support operating on multiple tuples at once, but I was wondering how performance would be if we could.
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/arrow-converter.cc@231
PS35, Line 231: SlotWriter::init_status_ = Status(ERROR_ARROW_IMPALA_TYPE_MISMATCH);
If we hit this branch, we shouldn't try to use timestamp_array_ at line 235. Should probably also add a DCHECK() that init_status_.ok() in methods relying on init success.
In other code we usually have a separate
Status Init()
method to handle initialization that may fail. That would be a more familiar pattern to use.
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/paimon/paimon-jni-scan-node.cc
File be/src/exec/paimon/paimon-jni-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/paimon/paimon-jni-scan-node.cc@a186
PS35, Line 186:
This check was dropped, which can lead to calling jni_scanner_ when it's uninitialized.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 35
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 18:42:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-added some paimon query tests for edge cases previously not covered
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Assisted-by: Claude Opus 5.5 (GitHub Copilot)
Reviewed-on: http://gerrit.cloudera.org:8080/24190
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
M be/src/testutil/desc-tbl-builder.cc
M be/src/testutil/desc-tbl-builder.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
19 files changed, 1,953 insertions(+), 513 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 46
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 37:
Several bugs and issues found.
Marking this as WIP until tests are extended to cover them and they are fixed.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 37
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 11:37:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello ji chen, Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#43).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-added some paimon query tests for edge cases previously not covered
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Assisted-by: Claude Opus 5.5 (GitHub Copilot)
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
M be/src/testutil/desc-tbl-builder.cc
M be/src/testutil/desc-tbl-builder.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
19 files changed, 1,952 insertions(+), 513 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/43
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 43
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22282/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Apr 2026 13:09:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#11).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 669 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/11
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 11
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#16).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 1,088 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/16
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 16
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#29).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
12 files changed, 1,239 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/29
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 29
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 26:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23344/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 26
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 11:00:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#26).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
12 files changed, 1,243 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/26
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 26
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 24:
Patch 24 added arrow array conversion to an array of raw values (i.e. void**) and necessary refactoring.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 24
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Jul 2026 11:24:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 40: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc
File be/src/exec/paimon/paimon-jni-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc@1
PS39, Line 1: // Licensed to the Apache Software Foundation (ASF) under one
> Done
Ack
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc@271
PS39, Line 271: scratch_batch_->ReleaseResources(nullptr);
> Can parent nodes still use data in the scratch batch?
Ack
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 40
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2026 15:25:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 40:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc
File be/src/exec/paimon/paimon-jni-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc@1
PS39, Line 1: // Licensed to the Apache Software Foundation (ASF) under one
> Excess whitespace here.
Done
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc@271
PS39, Line 271: scratch_batch_->ReleaseResources(nullptr);
> This call can wind up releasing resources that a parent is still using. Tak
Can parent nodes still use data in the scratch batch?
According to https://github.com/apache/impala/blob/master/be/src/exec/exec-node.h#L248
"It is only safe to call Close() on the child node while the parent node is still returning rows if the parent node fully materializes the child's input."
HdfsParquetScanner is different in that it is not an exec node.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 40
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2026 09:40:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#35).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
13 files changed, 1,384 insertions(+), 520 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/35
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 35
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "ji chen (Code Review)" <ge...@cloudera.org>.
ji chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 37:
Great thanks for the improvement, I have one question, can complex types also applied this solution in the later patch?
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 37
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 18:53:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 44:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24672/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 44
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Oct 2026 13:47:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#14).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 830 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/14
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 14
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#19).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Convert Impala row batches to Arrow record batches and vice versa.
Updated paimon jni reader to use this conversion when parsing arrow
records.
Current implementation doesn't support conversion of nested types, it
writes nulls instead.
TODO:
-split up into two parts:
arrow -> impala conversion
impala->arrow conversion
-handle nested types (in a separate patch)
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 1,653 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/19
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 19
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 38:
(2 comments)
Rewrote arrow-converter-test to handle all types and edge cases (like invalid timestamps, nulls, etc) and invalid cases (wrong descriptors, non-matching line numbers etc), and fixed all bugs found.
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/arrow-converter.cc@231
PS35, Line 231: DateSlotWriter(const SlotDescriptor* slot_desc, const arrow::Array* array) :
> If we hit this branch, we shouldn't try to use timestamp_array_ at line 235
Done
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/paimon/paimon-jni-scan-node.cc
File be/src/exec/paimon/paimon-jni-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/paimon/paimon-jni-scan-node.cc@a186
PS35, Line 186:
> This check was dropped, which can lead to calling jni_scanner_ when it's un
Good catch.
Fixed this, and added tests cases to paimon-query to handle more edge cases like this.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 38
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 11:19:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 40:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24404/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 40
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2026 10:03:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 31:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23498/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 31
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 04 Aug 2026 13:16:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 30:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23483/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 30
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 03 Aug 2026 10:30:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#28).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
13 files changed, 1,239 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/28
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 28
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 27:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@47
PS26, Line 47: set(tuple_mem, 0, num_rows * tupl
> probably it is more efficient to memset the whole array to 0
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@116
PS26, Line 116: return WriteSlotBatchImpl<true>(tuple_mem, tuple_size, start_row_idx, end_row_idx);
> The arrow array could be also checked if it ha any null values:
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@194
PS26, Line 194: return SetDecimalVal(data, byte_length, slot)
> Shouldn't the size match the size we expect exactly? The size should be cle
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@210
PS26, Line 210:
> Arrow has its own decimal32/64/128/256 types.
This would work, but we have arrow 15.0.0 which doesn't have decimal32 and 64 as they were added later.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@226
PS26, Line 226: Status WriteSlotInternal(int row_idx, void* slot) {
> This doesn't change per row, the final Timezone ptr could be set in timezon
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@230
PS26, Line 230: ::arrow::TimeUnit::NANO:
> This looks incorrect for negative values, because those are truncated towar
TimestampValue::FromUnixTimeNanos does call SplitTime though, which truncates it towards negative infinity.
There are tests in timestamp-test.cc which call FromUnixTimeNanos with a negative nanos value.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@246
PS26, Line 246:
> This lacks validation (unlike Parquet). Or validation is done elsewhere?
I added returning error status when encountering invalid values.
For internally produced arrow arrays, it will make sense to add them in the patch that converts impala batches to arrow.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@254
PS26, Line 254:
> Not sure if the argument is useful just to pass this in for the error messa
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@267
PS26, Line 267:
> If small enough then small strings could be created from the start.
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@268
PS26, Line 268: public SlotWriterImpl<StringSlotWriter<ArrowArrayType>
> Would it be possible to point to the arrow array to avoid the copy? I assum
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@318
PS26, Line 318: break;
: }
: case TYPE_STRING:
: case TYPE_VARCHAR: {
: if (type.IsBinaryType()) { // byte[]
: writer.reset(new StringSlotWriter<arrow::BinaryArray>(slot_desc, array));
: } else {
: writer.reset(new StringSlotWriter<arrow::StringArray>(slot_desc, array));
: }
: break;
: }
: case TYPE_BOOLEAN: {
: writer.reset(new BaseSlotWriter<bool, arrow::BooleanArray>(slot_desc, array));
: break;
: }
> This looks strange - VARCHAR is written by StringSlotWriter, while CHAR is
Done
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 27
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 03:01:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 29:
Patch 29 is a rebase.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 29
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 10:23:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 25:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23228/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 25
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 14 Jul 2026 12:54:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 44: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 44
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Oct 2026 07:42:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 31:
(2 comments)
Also fixed a bug with the paimon scanner loop not exiting when reaching a LIMIT.
http://gerrit.cloudera.org:8080/#/c/24190/29/be/src/exec/arrow-converter.h
File be/src/exec/arrow-converter.h:
http://gerrit.cloudera.org:8080/#/c/24190/29/be/src/exec/arrow-converter.h@49
PS29, Line 49: RuntimeState* state);
> How about adding a `uint8_t* value_buffer` parameter here so that we don't
Done
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@268
PS26, Line 268: public SlotWriterImpl<StringSlotWriter<ArrowArrayType>
> Done
This didn't work.
During paimon scan, a batch of arrow data is read at once and then processed.
The problem is that requesting a new batch can happen while some rows in the current row batch still correspond to the previous batch.
So I reintroduced the allocate and copy (except for smallable strings).
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 31
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 04 Aug 2026 12:56:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 38:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/24346/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 38
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 11:47:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 36:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24279/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 36
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 11:06:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello ji chen, Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#44).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-added some paimon query tests for edge cases previously not covered
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Assisted-by: Claude Opus 5.5 (GitHub Copilot)
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
M be/src/testutil/desc-tbl-builder.cc
M be/src/testutil/desc-tbl-builder.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
19 files changed, 1,953 insertions(+), 513 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/44
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 44
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#15).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 954 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/15
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 15
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#18).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 1,229 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/18
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 18
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#10).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 612 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/10
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 10
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#20).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
12 files changed, 1,129 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/20
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 20
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#3).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/paimon/paimon-jni-row-reader.cc
M be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
5 files changed, 133 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/3
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 3
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#30).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
12 files changed, 1,239 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/30
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 30
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22136/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 2
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Apr 2026 09:08:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 18:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22330/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 18
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 May 2026 13:44:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22151/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 3
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Apr 2026 11:57:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#13).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 750 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/13
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 13
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 14:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22269/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 14
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Apr 2026 10:13:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 20:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22483/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 20
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 10:48:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22187/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 6
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Apr 2026 10:54:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#32).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
12 files changed, 1,256 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/32
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 32
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 29:
(1 comment)
Hi!
http://gerrit.cloudera.org:8080/#/c/24190/29/be/src/exec/arrow-converter.h
File be/src/exec/arrow-converter.h:
http://gerrit.cloudera.org:8080/#/c/24190/29/be/src/exec/arrow-converter.h@49
PS29, Line 49:
How about adding a `uint8_t* value_buffer` parameter here so that we don't need to allocate a new buffer for each call?
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 29
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 03 Aug 2026 08:00:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 34:
Patch 34 is a rebase.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 34
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Sep 2026 08:17:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 24:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23164/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 24
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Jul 2026 11:46:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 35: Code-Review+1
> Patch Set 35:
>
> (4 comments)
>
> Thanks for the comments.
> There were also some minor issues in arrow-converter-benchmark.cc that I fixed.
Thanks!
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 35
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Sep 2026 07:39:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 36:
Patch set 36 is a rebase.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 36
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 10:43:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 39:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24347/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 39
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 12:45:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 26:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter-test.cc
File be/src/exec/arrow-converter-test.cc:
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter-test.cc@110
PS24, Line 110: EXPECT_EQ(value->Len(), (int)original_value.siz
> nit: comparing int to size_t, probably worth an explicit cast
Done
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@84
PS24, Line 84: slot_desc_(slot_desc) {
> A potential optimization could be to optimize for cases with no NULLs to av
Done
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@89
PS24, Line 89: int end_row_idx) {
> It may be faster to set all tuples to 0 (meaning non-null) and only set nul
Done
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@138
PS24, Line 138:
> Can you add a comment about these types? Or a more descriptive name could h
Done
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@214
PS24, Line 214:
> Some of this code could be possibly merged with Parquet scanning code.
ParquetTimestampDecoder seems too Parquet specific to be used here.
For example, there is arrow::TimeUnit::SECOND unit that I don't see in Parquet, adding that there would affect Parquet scanning.
So I found it best to keep this separate.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@311
PS24, Line 311:
> Missing break; after this!
Done
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@377
PS24, Line 377:
> Does arrow store timestamps as UTC? Iceberg uses mainly timestamps without
Arrow has a "timezone" type parameter for a Timestamp type, I haven't found much information about it (like what empty value means).
I kept the timezone handling as it was in the previous arrow scanning in paimon-jni-row-reader.cc
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@407
PS24, Line 407:
> Unaligned allocation will cause issues with larger types.
Done
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@417
PS24, Line 417: if (UNLIKELY(value_buffer == nul
> This will crash in case of complex types. Better to return a status with an
Done
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 26
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 10:36:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 24:
(4 comments)
Thanks for the review!
I found a couple more issues, will resolve all together.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter-test.cc
File be/src/exec/arrow-converter-test.cc:
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter-test.cc@110
PS24, Line 110: EXPECT_EQ(value->Len(), original_value.size());
nit: comparing int to size_t, probably worth an explicit cast
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@311
PS24, Line 311: writer.reset(new VarCharSlotWriter(slot_desc, array, type.len));
Missing break; after this!
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@407
PS24, Line 407: Unaligned
Unaligned allocation will cause issues with larger types.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@417
PS24, Line 417: DCHECK(writer.get() != nullptr);
This will crash in case of complex types. Better to return a status with an error message (e.g. complex types are not supported)
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 24
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Jul 2026 10:36:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#25).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
12 files changed, 1,233 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/25
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 25
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#31).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
12 files changed, 1,256 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/31
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 31
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 17:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22320/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 17
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 May 2026 06:37:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#2).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/paimon/paimon-jni-row-reader.cc
M be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
5 files changed, 139 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/2
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 2
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 21:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22565/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 21
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 May 2026 06:07:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22215/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 9
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Apr 2026 07:39:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#34).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
12 files changed, 1,308 insertions(+), 200 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/34
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 34
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 34:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23962/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 34
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Sep 2026 08:41:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 39:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/35/be/src/exec/arrow-converter.cc@231
PS35, Line 231: DateSlotWriter(const SlotDescriptor* slot_desc, const arrow::Array* array) :
> Done
Ack
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc
File be/src/exec/paimon/paimon-jni-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc@1
PS39, Line 1: // Licensed to the Apache Software Foundation (ASF) under one
Excess whitespace here.
http://gerrit.cloudera.org:8080/#/c/24190/39/be/src/exec/paimon/paimon-jni-scan-node.cc@271
PS39, Line 271: scratch_batch_->ReleaseResources(nullptr);
This call can wind up releasing resources that a parent is still using. Take a look at https://github.com/apache/impala/blob/master/be/src/exec/parquet/hdfs-parquet-scanner.cc#L314-L329 for an example how to handle it.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 39
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 15:45:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello ji chen, Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#40).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-added some paimon query tests for edge cases previously not covered
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Assisted-by: Claude Opus 5.5 (GitHub Copilot)
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
M be/src/testutil/desc-tbl-builder.cc
M be/src/testutil/desc-tbl-builder.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
19 files changed, 1,843 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/40
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 40
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22178/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 5
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Apr 2026 14:07:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#21).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
12 files changed, 1,129 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/21
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 21
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 31:
(2 comments)
> Patch Set 31:
>
> (2 comments)
>
> Also fixed a bug with the paimon scanner loop not exiting when reaching a LIMIT.
Thanks!
http://gerrit.cloudera.org:8080/#/c/24190/31/be/src/exec/arrow-converter.h
File be/src/exec/arrow-converter.h:
http://gerrit.cloudera.org:8080/#/c/24190/31/be/src/exec/arrow-converter.h@48
PS31, Line 48: void** raw_values, uint8_t* value_buffer, const ColumnType& type, MemPool* mem_pool,
How about returning an error if the type of the `array` argument does not match the `type` argument?
http://gerrit.cloudera.org:8080/#/c/24190/29/be/src/exec/arrow-converter.h
File be/src/exec/arrow-converter.h:
http://gerrit.cloudera.org:8080/#/c/24190/29/be/src/exec/arrow-converter.h@49
PS29, Line 49: RuntimeState* state);
> Done
Ack
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 31
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Aug 2026 03:01:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 35:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23976/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 35
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Sep 2026 10:50:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 33:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24190/31/be/src/exec/arrow-converter.h
File be/src/exec/arrow-converter.h:
http://gerrit.cloudera.org:8080/#/c/24190/31/be/src/exec/arrow-converter.h@48
PS31, Line 48: void** raw_values, uint8_t* value_buffer, const ColumnType& type, MemPool* mem_pool,
> How about returning an error if the type of the `array` argument does not m
I added an error status for this and a test case in arrow-converter-test.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 33
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 11:10:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 26:
(12 comments)
Still haven't processed the tests.
Most comments are about timestamps - these look preexisting issues with timestamp handling in Paimon (which seems minimally tests) Probably they could be moved to a follow up ticket.
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/24/be/src/exec/arrow-converter.cc@377
PS24, Line 377:
> Arrow has a "timezone" type parameter for a Timestamp type, I haven't found
This area needs some more comments - currently it is not explained what's actually happening.
const std::string& tz_name = type.timezone();
const Timezone* tz = tz_name.empty() ? UTCPTR : timezone_;
The tz_name non empty case fits timestamp with local timezone semantics, so the timestamp is interpreted as UTC, and converted into Impala's local timezone during scanning.
The empty case means timezone-agnostic (in Arrow they use name "timestamp naive"), which means that we assume that the data is not written as offset in local timezone and requires no conversion to local timezone.
If arrow conversion will be used internally in Impala and support round-trips between Impala's RowBatch and arrow, then timezone will need to be empty to avoid extra conversions. Setting timezone is only ok during scanning to convert from utc to local.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@47
PS26, Line 47: uple->ClearNullBits(*tuple_desc);
probably it is more efficient to memset the whole array to 0
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@116
PS26, Line 116: if (slot_desc_->is_nullable()) {
The arrow array could be also checked if it ha any null values:
see HasValidityBitmap and MayHaveLogicalNulls in https://arrow.apache.org/docs/cpp/api/array.html
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@194
PS26, Line 194: DCHECK(byte_length > 0 && byte_length <= 16);
Shouldn't the size match the size we expect exactly? The size should be clear from the Impala type used.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@210
PS26, Line 210: binary_array_
Arrow has its own decimal32/64/128/256 types.
https://arrow.apache.org/docs/cpp/api/datatype.html
I assume Paimon returns BinaryArray, but if we do internal conversions, it would be better to use these types.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@226
PS26, Line 226: const Timezone* tz = tz_name.empty() ? UTCPTR : timezone_;
This doesn't change per row, the final Timezone ptr could be set in timezone_ in the constructor. It may also make sense to have have two different classes for Timestamps with or without timezone.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@230
PS26, Line 230: value / NANOS_PER_SEC, value % NANOS_PER_SEC
This looks incorrect for negative values, because those are truncated towards zero. See UtcFromUnixTimeLimitedRangeNanos for how this is handled in Parquet.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@246
PS26, Line 246: return Status::OK();
This lacks validation (unlike Parquet). Or validation is done elsewhere?
See ScalarColumnReader<TimestampValue, parquet::Type::INT96, true>::ValidateValue(
TimestampValue* val)
For validation probably the best would be to differentiate between internally and externally produced arrow arrays - for the external case we should validate if timestamps are in the valid range, while for internal case it should be a DCHECK.
Based on Paimon docs they allow timestamps in range 0000-01-01 00:00:00.000000000 to 9999-12-31 23:59:59.999999999 - Impala only allows timestamps from 1400.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@254
PS26, Line 254: IS_BINARY
Not sure if the argument is useful just to pass this in for the error message. The column name would be much more useful info.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@267
PS26, Line 267: v.size();
If small enough then small strings could be created from the start.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@268
PS26, Line 268: // Allocate memory and copy the bytes to the RowBatch.
Would it be possible to point to the arrow array to avoid the copy? I assume that in the Paimon case the arrow version will be kept in memory while the row batch is processed.
http://gerrit.cloudera.org:8080/#/c/24190/26/be/src/exec/arrow-converter.cc@318
PS26, Line 318: case TYPE_CHAR: {
: writer.reset(new VarCharSlotWriter(slot_desc, array, type.len));
: break;
: }
: case TYPE_STRING:
: case TYPE_VARCHAR: {
: if (type.IsBinaryType()) { // byte[]
: writer.reset(new StringSlotWriter<true, arrow::BinaryArray>(slot_desc, array,
: mem_pool));
: } else {
: writer.reset(new StringSlotWriter<false, arrow::StringArray>(slot_desc, array,
: mem_pool));
: }
: break;
: }
This looks strange - VARCHAR is written by StringSlotWriter, while CHAR is written by VarCharSlotWriter. Shouldn't it be called CharSlotWriter?
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 26
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 13:27:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 29:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23473/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 29
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 10:45:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Xuebin Su, jichen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#27).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
13 files changed, 1,239 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/27
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 27
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello ji chen, Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#39).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-added some paimon query tests for edge cases previously not covered
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Assisted-by: Claude Opus 5.5 (GitHub Copilot)
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
M be/src/testutil/desc-tbl-builder.cc
M be/src/testutil/desc-tbl-builder.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
19 files changed, 1,844 insertions(+), 516 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/39
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 39
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 44:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24190/43/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/43/be/src/exec/arrow-converter.cc@358
PS43, Line 358: // TODO: This truncates in bytes, not in UTF-8 characters. See IMPALA-5675.
> An issue here is that this can truncates in bytes and not characters.
Done
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 44
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Oct 2026 13:25:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 45: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 45
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Oct 2026 12:20:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 45:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14329/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 45
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Oct 2026 07:42:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22260/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 12
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 28 Apr 2026 09:12:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#12).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
10 files changed, 787 insertions(+), 515 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/12
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 12
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 13:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22268/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 13
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Apr 2026 09:00:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 19:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22445/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 19
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 May 2026 11:45:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 11:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22248/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 11
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 27 Apr 2026 14:51:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 4:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22177/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 4
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Apr 2026 09:34:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22204/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 8
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Apr 2026 11:18:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#6).
Change subject: WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
......................................................................
WIP IMPALA-14882: Add efficient conversion between RowBatch and arrow
Work In Progress
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
---
M be/src/exec/paimon/paimon-jni-row-reader.cc
M be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
5 files changed, 402 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/6
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 6
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: WIP IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 37:
Yes, complex types are handled too, but I separated it to another patch in https://gerrit.cloudera.org/#/c/24538/
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 37
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 05:58:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Hello ji chen, Xuebin Su, jichen, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24190
to look at the new patch set (#38).
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Added efficient conversion from an arrow record batch to a batch of
impala tuples.
Previously, there existed a conversion in paimon jni reader that
converted arrow arrays row-by-row. The new conversion writes all rows
per column (i.e. all elements of an arrow array into the corresponding
slots in the impala tuples) for each column/field/slot.
Fixed size parts of tuples have to be pre-allocated in a contiguous
memory, so that columnar writes can be efficient (i.e. a tuple's memory
can be calculated as base_ptr + index * tuple_size).
Paimon jni scan node was modified to convert the arrow record first into
a scratch batch, then apply filtered rows to the output batch, similarly
to how file readers work (e.g. HdfsParquetScanner).
This patch does not handle nested types (array/list, struct, map).
Testing:
-added arrow-converter-test.cc with tests for converting different types
-added some paimon query tests for edge cases previously not covered
-ran paimon tests locally
Measurement:
-added arrow-converter-benchmark for converting different types
Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Assisted-by: Claude Opus 5.5 (GitHub Copilot)
---
M be/src/benchmarks/CMakeLists.txt
A be/src/benchmarks/arrow-converter-benchmark.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/arrow-converter-test.cc
A be/src/exec/arrow-converter.cc
A be/src/exec/arrow-converter.h
M be/src/exec/paimon/CMakeLists.txt
D be/src/exec/paimon/paimon-jni-row-reader.cc
D be/src/exec/paimon/paimon-jni-row-reader.h
M be/src/exec/paimon/paimon-jni-scan-node.cc
M be/src/exec/paimon/paimon-jni-scan-node.h
M be/src/exec/scratch-tuple-batch.h
M be/src/runtime/string-value.h
M be/src/testutil/desc-tbl-builder.cc
M be/src/testutil/desc-tbl-builder.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
19 files changed, 1,842 insertions(+), 516 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24190/38
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 38
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 33:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23863/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 33
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 11:29:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 33:
(4 comments)
Leaving a few comments, I still want to go through the bulk of arrow-converter.cc in more detail.
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc
File be/src/benchmarks/arrow-converter-benchmark.cc:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc@48
PS33, Line 48: // Mixed 180 182 183 0.248X 0.248X 0.248X
This primarily shows Mixed is slower.
How does this compare to copying a tuple batch or an arrow record batch? I'm looking for an upper-bound on performance that might serve as a goal.
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc@116
PS33, Line 116:
nit: excess whitespace
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/arrow-converter.cc@54
PS33, Line 54: RETURN_IF_ERROR(WriteSlotBatch(arr.get(), 0, num_rows, slot_desc,
Have you benchmarked operating on a batch of tuple_mems and writing a column at a time?
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/paimon/CMakeLists.txt
File be/src/exec/paimon/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/paimon/CMakeLists.txt@a25
PS33, Line 25:
Should the file itself also be removed?
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 33
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Sep 2026 23:12:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Xuebin Su (Code Review)" <ge...@cloudera.org>.
Xuebin Su has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 40: Code-Review+1
(1 comment)
> Patch Set 40:
>
> (2 comments)
Thanks!
http://gerrit.cloudera.org:8080/#/c/24190/40/be/src/exec/arrow-converter-test.cc
File be/src/exec/arrow-converter-test.cc:
http://gerrit.cloudera.org:8080/#/c/24190/40/be/src/exec/arrow-converter-test.cc@49
PS40, Line 49: EXPECT_TRUE(_arrow_status.ok()) << _arrow_status.ToString(); \
nit: The backslashes are not aligned.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 40
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Oct 2026 09:01:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 35:
(4 comments)
Thanks for the comments.
There were also some minor issues in arrow-converter-benchmark.cc that I fixed.
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc
File be/src/benchmarks/arrow-converter-benchmark.cc:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc@48
PS33, Line 48: // Mixed 271 277 281 0.238X 0.237X 0.234X
> This primarily shows Mixed is slower.
Added an arrow to arrow copy suite for comparison.
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/benchmarks/arrow-converter-benchmark.cc@116
PS33, Line 116: const std::shared_ptr<arrow::ArrayData>& array_data =
> nit: excess whitespace
Done
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/arrow-converter.cc@54
PS33, Line 54: RETURN_IF_ERROR(WriteSlotBatch(arr.get(), 0, num_rows, slot_desc,
> Have you benchmarked operating on a batch of tuple_mems and writing a colum
I don't understand what you mean exactly.
Columns are written one by one during this conversion into a pre-allocated tuple_mem.
The arrow-converter-benchmark.cc calls this same conversion.
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/paimon/CMakeLists.txt
File be/src/exec/paimon/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/24190/33/be/src/exec/paimon/CMakeLists.txt@a25
PS33, Line 25:
> Should the file itself also be removed?
Done
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 35
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Sep 2026 10:26:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 45: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 45
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Oct 2026 07:42:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 43:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24577/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 43
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Oct 2026 10:47:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Balazs Hevele (Code Review)" <ge...@cloudera.org>.
Balazs Hevele has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 43:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/benchmarks/arrow-converter-benchmark.cc
File be/src/benchmarks/arrow-converter-benchmark.cc:
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/benchmarks/arrow-converter-benchmark.cc@48
PS42, Line 48: Mixed
> nit, ok to move to followup commit:
Ack
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/exec/arrow-converter.cc@163
PS42, Line 163: Status CastArray(const ArrowArrayType** result) cons
> shouldn't we subtract start_row_idx here, like in line 116? (thx claude for
Done, good catch
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/exec/arrow-converter.cc@270
PS42, Line 270: void SetDecimalVal(const uint8_t* buffer,
> Can this come from corrupt data, or is a program error if this happens? In
This can technically come from corrupt data -,though it's not a realistic case with data coming from Paimon-, so keeping this as a Status.
On the other hand, the check in SetDecimalVal is a program error now, so changed that one to a DCHECK.
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/exec/arrow-converter.cc@275
PS42, Line 275: }
> Won't this allocate on each call - why no use a static const in class or co
Done
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/exec/arrow-converter.cc@319
PS42, Line 319:
> nit: this can overflow for corrupt values, leading to potentially accepting
Done
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/exec/arrow-converter.cc@426
PS42, Line 426: writer->reset(
> What is the approach to varchar truncation? Can we assume that this is alre
Added the truncation to the conversion
http://gerrit.cloudera.org:8080/#/c/24190/42/be/src/exec/arrow-converter.cc@536
PS42, Line 536:
: Status ArrowToTupleConverter::Convert(cons
> It would be nicer to avoid the per row batch * column allocation. I don't s
Good idea.
Made a class from it that keeps the SlotWriters between conversions. Paimon now uses it. Added test cases for it.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 43
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Oct 2026 10:23:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24190 )
Change subject: IMPALA-14882: part1: Convert arrow record batch to impala tuple batch
......................................................................
Patch Set 43: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24190/43/be/src/exec/arrow-converter.cc
File be/src/exec/arrow-converter.cc:
http://gerrit.cloudera.org:8080/#/c/24190/43/be/src/exec/arrow-converter.cc@358
PS43, Line 358: if (max_len_ >= 0) v = v.substr(0, max_len_);
An issue here is that this can truncates in bytes and not characters.
This is a common problem in Impala, e.g. IMPALA-5675. A todo could be added.
--
To view, visit http://gerrit.cloudera.org:8080/24190
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea544b3c71d9211c893f0fec3527ebe84155ebcd
Gerrit-Change-Number: 24190
Gerrit-PatchSet: 43
Gerrit-Owner: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Balazs Hevele <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Xuebin Su <xs...@cloudera.com>
Gerrit-Reviewer: ji chen <ji...@163.com>
Gerrit-Reviewer: jichen <ji...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 13:49:05 +0000
Gerrit-HasComments: Yes