You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhouyuan (via GitHub)" <gi...@apache.org> on 2025/08/29 14:19:04 UTC

[PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

zhouyuan opened a new pull request, #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585

   <!--
   Thank you for submitting a pull request! Here are some tips:
   
   1. For first-time contributors, please read our contributing guide:
      https://github.com/apache/incubator-gluten/blob/main/CONTRIBUTING.md
   2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
   3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
      Velox or ClickHouse backend, respectively.
   4. If the PR is not ready for review, please mark it as a draft.
   -->
   
   ## What changes are proposed in this pull request?
   
   This patch adds a refactor on hive connector registration. instead of registering when instancing Velox, we defer it to when creating the wholestage context. This allows us to get the session level configurations, and pass to hive/velox 
   
   <!--
   Provide a clear and concise description of the changes introduced in this PR.
   Ensure the PR description aligns with the code changes, especially after updates.
   If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
   when the PR is merged.
   -->
   
   ## How was this patch tested?
   local verified
   
   <!--
   Describe how the changes were tested, if applicable.
   Include new tests to validate the functionality, if necessary.
   For UI-related changes, attach screenshots to demonstrate the updates.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10585: [GLUTEN-10113][VL] Refactor hive connector register to accept session level config
URL: https://github.com/apache/incubator-gluten/pull/10585


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on code in PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#discussion_r2315329654


##########
cpp/velox/compute/VeloxBackend.cc:
##########
@@ -294,6 +293,12 @@ void VeloxBackend::initCache() {
 }
 
 void VeloxBackend::initConnector(const std::shared_ptr<velox::config::ConfigBase>& hiveConf) {
+  for (const auto& [key, value] : hiveConf->rawConfigs()) {
+    // always update to use new session level conf
+    backendConf_->set(key, value);

Review Comment:
   If the `initConnector` is only called from `WholeStageResultIterator`, is it possible to remove the filesystem related configurations from the backend conf?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#discussion_r2319250500


##########
cpp/velox/compute/VeloxBackend.cc:
##########
@@ -294,15 +293,24 @@ void VeloxBackend::initCache() {
 }
 
 void VeloxBackend::initConnector(const std::shared_ptr<velox::config::ConfigBase>& hiveConf) {

Review Comment:
   Should we move the function to WholeStageResultIterator?



##########
cpp/velox/compute/VeloxBackend.cc:
##########
@@ -91,8 +91,8 @@ void veloxRuntimeReleaser(Runtime* runtime) {
 void VeloxBackend::init(
     std::unique_ptr<AllocationListener> listener,
     const std::unordered_map<std::string, std::string>& conf) {
-  backendConf_ =
-      std::make_shared<facebook::velox::config::ConfigBase>(std::unordered_map<std::string, std::string>(conf));
+  backendConf_ = std::make_shared<facebook::velox::config::ConfigBase>(

Review Comment:
   Could we generate the HiveConfig by veloxCfg_ in WholeStageResultIterator every time?



##########
cpp/velox/compute/VeloxBackend.cc:
##########
@@ -91,8 +91,8 @@ void veloxRuntimeReleaser(Runtime* runtime) {
 void VeloxBackend::init(
     std::unique_ptr<AllocationListener> listener,
     const std::unordered_map<std::string, std::string>& conf) {
-  backendConf_ =
-      std::make_shared<facebook::velox::config::ConfigBase>(std::unordered_map<std::string, std::string>(conf));
+  backendConf_ = std::make_shared<facebook::velox::config::ConfigBase>(

Review Comment:
   mutable true make copy the configs when iterating the configs



##########
cpp/velox/compute/WholeStageResultIterator.cc:
##########
@@ -102,6 +103,10 @@ WholeStageResultIterator::WholeStageResultIterator(
   fileSystem->mkdir(spillDir);
   task_->setSpillDirectory(spillDir);
 
+  // register the hive connectors
+  std::call_once(

Review Comment:
   In what case, we may register twice?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on code in PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#discussion_r2315323004


##########
cpp/velox/compute/WholeStageResultIterator.cc:
##########
@@ -102,6 +103,10 @@ WholeStageResultIterator::WholeStageResultIterator(
   fileSystem->mkdir(spillDir);
   task_->setSpillDirectory(spillDir);
 
+  // register the hive connectors
+  std::call_once(
+      gluten::VeloxBackend::get()->regFlag, [&]() { gluten::VeloxBackend::get()->initConnector(veloxCfg_); });

Review Comment:
   Wonder if we could register with a unique connector ID for each `WholeStageResultIterator` and call `velox::connector::unregisterConnector` from the dtor?
   
   The connector ID can use similar naming as the queryID
   
   ```
   fmt::format(
             "Gluten_Stage_{}_TID_{}_VTID_{}",
             std::to_string(taskInfo_.stageId),
             std::to_string(taskInfo_.taskId),
             std::to_string(taskInfo_.vId))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#discussion_r2321469085


##########
cpp/velox/compute/VeloxBackend.cc:
##########
@@ -294,6 +293,12 @@ void VeloxBackend::initCache() {
 }
 
 void VeloxBackend::initConnector(const std::shared_ptr<velox::config::ConfigBase>& hiveConf) {
+  for (const auto& [key, value] : hiveConf->rawConfigs()) {
+    // always update to use new session level conf
+    backendConf_->set(key, value);

Review Comment:
   Users may pass/modify the filesystem related configurations at session level, by setting the keys here it will allow us to pick the latest values



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#discussion_r2418395026


##########
cpp/velox/compute/WholeStageResultIterator.cc:
##########
@@ -102,6 +103,10 @@ WholeStageResultIterator::WholeStageResultIterator(
   fileSystem->mkdir(spillDir);
   task_->setSpillDirectory(spillDir);
 
+  // register the hive connectors
+  std::call_once(
+      gluten::VeloxBackend::get()->regFlag, [&]() { gluten::VeloxBackend::get()->initConnector(veloxCfg_); });

Review Comment:
   it looks like in Spark case, sometimes two stages will be scheduled to run at the same time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "xinghuayu007 (via GitHub)" <gi...@apache.org>.
xinghuayu007 commented on code in PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#discussion_r2315288059


##########
cpp/velox/utils/ConfigExtractor.cc:
##########
@@ -281,7 +281,7 @@ std::shared_ptr<facebook::velox::config::ConfigBase> getHiveConfig(
   hiveConfMap[facebook::velox::connector::hive::HiveConfig::kParquetUseColumnNames] = "true";
   hiveConfMap[facebook::velox::connector::hive::HiveConfig::kOrcUseColumnNames] = "true";
 
-  return std::make_shared<facebook::velox::config::ConfigBase>(std::move(hiveConfMap));
+  return std::make_shared<facebook::velox::config::ConfigBase>(std::move(hiveConfMap), false/*mutable*/);

Review Comment:
   The default value of parameter 'mutable' is false, no need to pass it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#discussion_r2321463485


##########
cpp/velox/compute/VeloxBackend.cc:
##########
@@ -91,8 +91,8 @@ void veloxRuntimeReleaser(Runtime* runtime) {
 void VeloxBackend::init(
     std::unique_ptr<AllocationListener> listener,
     const std::unordered_map<std::string, std::string>& conf) {
-  backendConf_ =
-      std::make_shared<facebook::velox::config::ConfigBase>(std::unordered_map<std::string, std::string>(conf));
+  backendConf_ = std::make_shared<facebook::velox::config::ConfigBase>(

Review Comment:
   No, here `veloxCfg_` contains only session level configurations, it does not cover the configurations defined in spark.conf



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#issuecomment-3698072887

   This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#issuecomment-3237200306

   https://github.com/apache/incubator-gluten/issues/10113


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-10113][VL] Refactor hive connector register to accept session level config [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10585:
URL: https://github.com/apache/incubator-gluten/pull/10585#issuecomment-3726742955

   This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org