You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "KevinyhZou (via GitHub)" <gi...@apache.org> on 2026/10/16 03:08:06 UTC
[PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
KevinyhZou opened a new pull request, #13334:
URL: https://github.com/apache/gluten/pull/13334
#13256
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
when the PR is merged.
-->
support read pamin table in clickhouse backend
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6207152446
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6357924702
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6213390276
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6461399292
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6207981758
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #13334:
URL: https://github.com/apache/gluten/pull/13334#discussion_r4420429313
##########
backends-clickhouse/src-paimon/main/scala/org/apache/gluten/component/CHPaimonComponent.scala:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.component
+
+import org.apache.gluten.backendsapi.clickhouse.CHBackend
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.extension.columnar.heuristic.HeuristicTransform
+import org.apache.gluten.extension.columnar.validator.Validators
+import org.apache.gluten.extension.injector.Injector
+
+import org.apache.spark.util.SparkReflectionUtil
+
+class CHPaimonComponent extends Component {
Review Comment:
Could we add a short comment immediately above `CHPaimonComponent` explaining how it is registered? In particular, `src-paimon/main/resources/META-INF/gluten-components/org.apache.gluten.component.CHPaimonComponent` is an intentionally empty marker: Gluten discovers the component by the resource **file name** when the `paimon` profile packages it, and does not read the file contents. That connection is not apparent from this class or the new marker, so a comment here would help prevent someone from deleting the seemingly unused empty file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6360754719
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6365430232
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6207452809
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6357310331
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6416589947
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6439251779
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6357304092
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6464036382
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6440256516
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #13334:
URL: https://github.com/apache/gluten/pull/13334#discussion_r4420406534
##########
cpp-ch/local-engine/Storages/SubstraitSource/Paimon/HdfsPaimonFileSystem.cpp:
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "HdfsPaimonFileSystem.h"
+
+#include <Interpreters/Context.h>
+#include <Storages/ObjectStorage/HDFS/HDFSCommon.h>
+#include <Common/logger_useful.h>
+#include <Poco/URI.h>
+#include <hdfs/hdfs.h>
+
+#include <algorithm>
+#include <cerrno>
+#include <chrono>
+#include <limits>
+#include <mutex>
+#include <utility>
+
+namespace local_engine::paimon
+{
+namespace
+{
+using Status = ::paimon::Status;
+
+struct HdfsConnection
+{
+ DB::HDFSFSPtr fs;
+ explicit HdfsConnection(const std::string & uri, DB::ContextPtr context)
+ {
+ auto builder = DB::createHDFSBuilder(uri, context->getConfigRef());
+ fs = DB::createHDFSFS(builder.get());
+ }
+};
+
+std::string pathPart(const std::string & uri)
+{
+ Poco::URI parsed(uri);
+ return parsed.getPath();
+}
+
+Status hdfsError(const std::string & operation, const std::string & path)
+{
+ return Status::IOError(operation, " ", path, ": ", hdfsGetLastError());
+}
+
+class HdfsStatus final : public ::paimon::FileStatus, public ::paimon::BasicFileStatus
+{
+ std::string path_;
+ int64_t size_;
+ int64_t modification_time_;
+ bool directory_;
+
+public:
+ explicit HdfsStatus(const hdfsFileInfo & info)
+ : path_(info.mName)
+ , size_(info.mSize)
+ , modification_time_(static_cast<int64_t>(info.mLastMod) * 1000)
+ , directory_(info.mKind == kObjectKindDirectory)
+ {
+ }
+
+ int64_t GetLen() const override { return size_; }
+ bool IsDir() const override { return directory_; }
+ std::string GetPath() const override { return path_; }
+ int64_t GetModificationTime() const override { return modification_time_; }
+};
+
+class HdfsInputStream final : public ::paimon::InputStream
+{
+ std::shared_ptr<HdfsConnection> connection_;
+ hdfsFile file_;
+ std::string uri_;
+ int64_t length_;
+ int64_t position_ = 0;
+ uint64_t read_calls_ = 0;
+ uint64_t read_bytes_ = 0;
+ uint64_t read_time_ns_ = 0;
+ mutable std::mutex mutex_;
+
+public:
+ HdfsInputStream(std::shared_ptr<HdfsConnection> connection, hdfsFile file, std::string uri, int64_t length)
+ : connection_(std::move(connection)), file_(file), uri_(std::move(uri)), length_(length) { }
+ ~HdfsInputStream() override
+ {
+ Close();
+ if (read_calls_ > 0)
+ LOG_INFO(
+ &Poco::Logger::get("PaimonHDFS"),
+ "HDFS read summary: path={}, calls={}, bytes={}, read_time_ms={}",
+ uri_,
+ read_calls_,
+ read_bytes_,
+ read_time_ns_ / 1000000);
+ }
+
+ Status Close() override
+ {
+ std::lock_guard lock(mutex_);
+ if (file_ && hdfsCloseFile(connection_->fs.get(), file_) != 0)
+ return hdfsError("close", uri_);
+ file_ = nullptr;
+ return Status::OK();
+ }
+
+ Status Seek(int64_t offset, ::paimon::SeekOrigin origin) override
+ {
+ std::lock_guard lock(mutex_);
+ if (origin != ::paimon::FS_SEEK_SET && origin != ::paimon::FS_SEEK_CUR && origin != ::paimon::FS_SEEK_END)
+ return Status::Invalid("Invalid HDFS seek origin: ", uri_);
+ int64_t base = origin == ::paimon::FS_SEEK_END ? length_ : origin == ::paimon::FS_SEEK_CUR ? position_ : 0;
+ if (offset < -base || offset > length_ - base)
+ return Status::Invalid("HDFS seek out of range: ", uri_);
+ position_ = base + offset;
+ return Status::OK();
+ }
+
+ ::paimon::Result<int64_t> GetPos() const override
+ {
+ std::lock_guard lock(mutex_);
+ return position_;
+ }
+
+ ::paimon::Result<int64_t> Read(char * buffer, int64_t size) override
+ {
+ std::lock_guard lock(mutex_);
+ auto result = readAt(buffer, size, position_);
+ if (result.ok())
+ position_ += result.value();
+ return result;
+ }
+
+ ::paimon::Result<int64_t> Read(char * buffer, int64_t size, int64_t offset) override
+ {
+ std::lock_guard lock(mutex_);
+ return readAt(buffer, size, offset);
+ }
+
+ void ReadAsync(char * buffer, int64_t size, int64_t offset, std::function<void(Status)> && callback) override
+ {
+ auto result = Read(buffer, size, offset);
+ callback(result.ok() ? Status::OK() : result.status());
+ }
+
+ ::paimon::Result<std::string> GetUri() const override { return uri_; }
+ ::paimon::Result<int64_t> Length() const override { return length_; }
+
+private:
+ ::paimon::Result<int64_t> readAt(char * buffer, int64_t size, int64_t offset)
+ {
+ if (!file_ || size < 0 || offset < 0)
+ return Status::Invalid("Invalid HDFS read: ", uri_);
+ if (offset > length_ || size > length_ - offset)
+ return Status::IOError("HDFS read exceeds file length: ", uri_);
+ int64_t total = 0;
+ while (total < size)
+ {
+ auto count = static_cast<tSize>(std::min<int64_t>(size - total, std::numeric_limits<tSize>::max()));
+ auto start = std::chrono::steady_clock::now();
+ auto read = hdfsPread(connection_->fs.get(), file_, buffer + total, count, offset + total);
+ read_time_ns_ += std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now() - start).count();
+ ++read_calls_;
+ if (read < 0)
+ return hdfsError("read", uri_);
+ if (read == 0)
+ return Status::IOError("Unexpected EOF reading HDFS file: ", uri_);
+ read_bytes_ += read;
+ total += read;
+ }
+ return total;
+ }
+};
+
+class HdfsPaimonFileSystem final : public ::paimon::FileSystem
+{
+ std::shared_ptr<HdfsConnection> connection_;
+
+public:
+ HdfsPaimonFileSystem(const std::string & table_path, DB::ContextPtr context)
+ {
+ Poco::URI uri(table_path);
+ connection_ = std::make_shared<HdfsConnection>(uri.getScheme() + "://" + uri.getAuthority(), context);
+ }
+
+ ::paimon::Result<std::unique_ptr<::paimon::InputStream>> Open(const std::string & path) const override
+ {
+ auto hdfs_path = pathPart(path);
+ auto * info = hdfsGetPathInfo(connection_->fs.get(), hdfs_path.c_str());
+ if (!info)
+ return hdfsError("stat", path);
+ int64_t length = info->mSize;
+ hdfsFreeFileInfo(info, 1);
+ auto file = hdfsOpenFile(connection_->fs.get(), hdfs_path.c_str(), O_RDONLY, 0, 0, 0);
+ if (!file)
+ return hdfsError("open", path);
+ return std::unique_ptr<::paimon::InputStream>(std::make_unique<HdfsInputStream>(connection_, file, path, length));
+ }
+
+ ::paimon::Result<bool> Exists(const std::string & path) const override
+ {
+ auto hdfs_path = pathPart(path);
+ if (hdfsExists(connection_->fs.get(), hdfs_path.c_str()) == 0)
+ return true;
+ if (errno == ENOENT)
Review Comment:
[P1] A missing HDFS path is not reliably recognized here. In libhdfs3, `hdfsExists()` calls `FileSystem::exist()`, which catches `FileNotFoundException` and returns `false`; `hdfsExists()` then returns `-1` without setting `errno`. If `errno` is zero or left over from another operation, this returns `IOError` instead of `false`. Paimon checks existence of optional paths (e.g. when looking up snapshots), so an ordinary missing path can fail a native HDFS scan. Could we handle the missing-path case without relying on `errno` after `hdfsExists()`, and cover `Exists(nonexistent_path)` in a test?
##########
ep/build-clickhouse/src/build-clickhouse.sh:
##########
@@ -17,6 +17,7 @@
export GLUTEN_SOURCE=$(builtin cd $(dirname $0)/../../..; pwd)
export CH_SOURCE_DIR=${GLUTEN_SOURCE}/cpp-ch/ClickHouse
CMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE:-RelWithDebInfo}
+GLUTEN_ENABLE_NATIVE_PAIMON=${GLUTEN_ENABLE_NATIVE_PAIMON:-OFF}
Review Comment:
[P2] This script defaults native Paimon to `OFF`, whereas the new ClickHouse.md instructions describe `-DGLUTEN_ENABLE_NATIVE_PAIMON=ON` as the default. Following the documented `-Ppaimon` JVM build and then invoking this build script without an environment override produces a Paimon-enabled JAR but a `libch.so` that rejects every native Paimon read with `Paimon native read is disabled`. Could we align the defaults or show the required `GLUTEN_ENABLE_NATIVE_PAIMON=ON` invocation explicitly in the docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6487754167
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6461105548
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6467935392
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #13334:
URL: https://github.com/apache/gluten/pull/13334#discussion_r4420415291
##########
gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto:
##########
@@ -251,6 +251,12 @@ message ReadRel {
}
message PaimonReadOptions {
+ message PaimonCppSplit {
Review Comment:
Could we give this message an implementation-neutral name? It carries a serialized Paimon DataSplit together with the table path, branch, and read options; nothing in the wire format is C++-specific. `PaimonCppSplit` (and `cpp_split` below) couples the shared Substrait protocol to the current reader implementation and may be misleading for other consumers. Something like `PaimonSplitReadSpec` / `split_read_spec` would describe the payload more accurately; the field number can remain 10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6213141769
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6207890617
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6361807307
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on code in PR #13334:
URL: https://github.com/apache/gluten/pull/13334#discussion_r4371434352
##########
gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesNode.java:
##########
@@ -44,5 +48,35 @@ public PaimonLocalFilesNode(
preferredLocations,
properties,
new ArrayList<>());
+ this.serializedSplits = null;
+ }
+
+ public PaimonLocalFilesNode(
+ Integer index, String tablePath, List<byte[]> splits, List<String> preferredLocations) {
+ super(
+ index,
+ java.util.Collections.nCopies(splits.size(), tablePath),
Review Comment:
Nit: could we add `import java.util.Collections;` and use `Collections.nCopies(...)` / `Collections.emptyMap()` throughout this constructor, consistent with the existing imports and usage of `ArrayList`, `List`, and `Map`?
##########
gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto:
##########
@@ -262,6 +262,9 @@ message ReadRel {
bool use_hive_split = 7;
bool raw_convertible = 8;
repeated string primary_keys = 9;
+ // Native Paimon reader input. The split is DataSplit.serialize() output.
+ string table_path = 10;
+ bytes serialized_split = 11;
Review Comment:
Could we preserve the branch and effective read-option overrides from the JVM scan when constructing the paimon-cpp read context?
Currently, only the table path and serialized split are passed through. The reader therefore defaults to the main branch, and query-specific option overrides are not forwarded. Reloading the persisted table options does not necessarily reproduce the JVM scan's read context.
I suggest grouping the SDK input into a dedicated `PaimonCppSplit` message inside `PaimonReadOptions`:
```proto
message PaimonCppSplit {
bytes serialized_data_split = 1;
string table_path = 2;
string branch = 3;
}
optional PaimonCppSplit cpp_split = 10;
```
Read-option overrides can use the existing properties mechanism, with the reader applying the branch and options via `WithBranch()` and `SetOptions()`. Please also add coverage for a non-main branch and a query-level read-option override.
##########
backends-clickhouse/src-paimon/main/scala/org/apache/gluten/execution/CHPaimonScanTransformer.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.gluten.exception.GlutenNotSupportException
+import org.apache.gluten.sql.shims.SparkShimLoader
+import org.apache.gluten.substrait.rel.{PaimonLocalFilesBuilder, SplitInfo}
+import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat
+
+import org.apache.spark.Partition
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, DynamicPruningExpression, Expression, Literal}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.read.Scan
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.paimon.CoreOptions
+import org.apache.paimon.io.DataOutputSerializer
+import org.apache.paimon.spark.{PaimonInputPartition, PaimonScan}
+import org.apache.paimon.table.{DataTable, FileStoreTable}
+import org.apache.paimon.table.source.DataSplit
+
+import scala.collection.JavaConverters._
+
+case class CHPaimonScanTransformer(
+ override val output: Seq[AttributeReference],
+ @transient override val scan: Scan,
+ override val runtimeFilters: Seq[Expression],
+ @transient override val table: Table,
+ override val keyGroupedPartitioning: Option[Seq[Expression]] = None,
+ override val commonPartitionValues: Option[Seq[(InternalRow, Int)]] = None,
+ override val pushDownFilters: Option[Seq[Expression]] = None)
+ extends BatchScanExecTransformerBase(
+ output = output,
+ scan = scan,
+ runtimeFilters = runtimeFilters,
+ table = table,
+ keyGroupedPartitioning = keyGroupedPartitioning,
+ commonPartitionValues = commonPartitionValues
+ ) {
+
+ private lazy val coreOptions: CoreOptions = scan match {
+ case paimonScan: PaimonScan =>
+ paimonScan.table match {
+ case dataTable: DataTable => dataTable.coreOptions()
+ case _ => throw new GlutenNotSupportException("Only support Paimon DataTable.")
+ }
+ case _ => throw new GlutenNotSupportException("Only support PaimonScan.")
+ }
+
+ override def getPartitionSchema: StructType = scan match {
+ case paimonScan: PaimonScan =>
+ val partitionKeys = paimonScan.table.partitionKeys()
+ StructType(scan.readSchema().filter(field => partitionKeys.contains(field.name)))
+ case _ => throw new GlutenNotSupportException("Only support PaimonScan.")
+ }
+
+ override def getDataSchema: StructType = scan match {
+ case paimonScan: PaimonScan =>
+ val partitionKeys = paimonScan.table.partitionKeys()
+ StructType(scan.readSchema().filterNot(field => partitionKeys.contains(field.name)))
+ case _ => throw new GlutenNotSupportException("Only support PaimonScan.")
+ }
+
+ override def withNewPushdownFilters(filters: Seq[Expression]): CHPaimonScanTransformer =
+ copy(pushDownFilters = Some(filters))
+
+ override def withOutput(newOutput: Seq[AttributeReference]): BatchScanExecTransformerBase =
+ copy(output = newOutput)
+
+ override lazy val fileFormat: ReadFileFormat = {
+ val formatStr = coreOptions.fileFormatString()
+ if ("parquet".equalsIgnoreCase(formatStr)) {
+ ReadFileFormat.ParquetReadFormat
+ } else if ("orc".equalsIgnoreCase(formatStr)) {
+ ReadFileFormat.OrcReadFormat
+ } else {
+ ReadFileFormat.UnknownFormat
+ }
+ }
+
+ override def doValidateInternal(): ValidationResult = {
+ scan match {
+ case paimonScan: PaimonScan =>
+ paimonScan.table match {
+ case _: FileStoreTable =>
+ ()
Review Comment:
Could we validate the supported split types before offloading this scan?
Checking only that the table is a `FileStoreTable` is not sufficient: Paimon 1.4.1 can produce `IncrementalSplit` for an incremental scan with `incremental-between-scan-mode=diff`. However, `getSplitInfosFromPartitions()` below only accepts `DataSplit` and throws for other split types. That method is called when generating the execution RDD, after plan validation, so this becomes a query failure rather than a JVM fallback.
Please reject unsupported split types or scan modes during validation and add a regression test verifying that an incremental diff scan falls back and returns the expected result. Supporting incremental reads natively can remain a follow-up.
##########
backends-clickhouse/src-paimon/main/scala/org/apache/gluten/component/CHOffloadPaimonScan.scala:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.component
+
+import org.apache.gluten.execution.{CHPaimonCountTransformer, CHPaimonScanTransformer, OffloadPaimonScan, PaimonScanTransformer}
+
+import org.apache.spark.sql.execution.{LocalTableScanExec, SparkPlan}
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+
+import org.apache.paimon.spark.read.PaimonLocalScan
+
+case object CHOffloadPaimonScan extends OffloadPaimonScan {
Review Comment:
Could `CHOffloadPaimonScan` extend `OffloadSingleNode` directly?
It completely overrides `offload()` and does not reuse the implementation in `OffloadPaimonScan`. The rule injection already accepts `OffloadSingleNode`, so inheriting from the concrete Paimon rule does not seem necessary. Extending the interface directly would also let us keep the existing `OffloadPaimonScan` case class unchanged, avoiding an unrelated change to the shared rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6469242369
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6490345536
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6496477026
all comments addressed, please help review the code, thanks. @liujiayi771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13256][CH]Support read paimon table in clickhouse backend by using `paimon-cpp` [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13334:
URL: https://github.com/apache/gluten/pull/13334#issuecomment-6496570574
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org