You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "lwz9103 (via GitHub)" <gi...@apache.org> on 2024/11/26 09:02:07 UTC
[PR] [GLUTEN-7028][CH] Support write parquet files with bucket [incubator-gluten]
lwz9103 opened a new pull request, #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052
## What changes were proposed in this pull request?
Support write parquet files with bucket
(Fixes: \#ISSUE-ID)
## How was this patch tested?
unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2500047095
https://github.com/apache/incubator-gluten/issues/7028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "lwz9103 (via GitHub)" <gi...@apache.org>.
lwz9103 commented on code in PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#discussion_r1867178920
##########
cpp-ch/local-engine/Storages/Output/NormalFileWriter.h:
##########
@@ -230,20 +230,57 @@ class WriteStats : public WriteStatsBase
struct FileNameGenerator
{
- const bool pattern;
- const std::string filename_or_pattern;
+ // Align with org.apache.spark.sql.execution.FileNamePlaceHolder
+ const std::vector<std::string> placeholders = {"{id}", "{bucket}"};
+ // Align with placeholders above
+ const std::vector<bool> need_replaced;
+ const std::string file_pattern;
+
+ FileNameGenerator(const std::string& file_pattern)
+ : file_pattern(file_pattern), need_replaced(compute_need_replaced(file_pattern))
+ {
+ }
+
+ std::vector<bool> compute_need_replaced(const std::string & file_pattern)
+ {
+ std::vector<bool> result;
+ for(const std::string& placeholder: placeholders)
+ {
+ if (file_pattern.find(placeholder) != std::string::npos)
+ result.push_back(true);
+ else
+ result.push_back(false);
+ }
+ return result;
+ }
+
+ std::string generate(const std::string bucket = "") const
+ {
+ std::string result = file_pattern;
+ if (need_replaced[0])
+ result = pattern_format(placeholders[0], toString(DB::UUIDHelpers::generateV4()));
+ if (need_replaced[1])
+ result = pattern_format(placeholders[1], bucket);
+ return result;
+ }
- std::string generate() const
+ std::string pattern_format(std::string arg, std::string replacement) const
Review Comment:
thanks, resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#discussion_r1866910322
##########
cpp-ch/local-engine/Storages/Output/NormalFileWriter.h:
##########
@@ -230,20 +230,57 @@ class WriteStats : public WriteStatsBase
struct FileNameGenerator
{
- const bool pattern;
- const std::string filename_or_pattern;
+ // Align with org.apache.spark.sql.execution.FileNamePlaceHolder
+ const std::vector<std::string> placeholders = {"{id}", "{bucket}"};
+ // Align with placeholders above
+ const std::vector<bool> need_replaced;
+ const std::string file_pattern;
+
+ FileNameGenerator(const std::string& file_pattern)
+ : file_pattern(file_pattern), need_replaced(compute_need_replaced(file_pattern))
+ {
+ }
+
+ std::vector<bool> compute_need_replaced(const std::string & file_pattern)
+ {
+ std::vector<bool> result;
+ for(const std::string& placeholder: placeholders)
+ {
+ if (file_pattern.find(placeholder) != std::string::npos)
+ result.push_back(true);
+ else
+ result.push_back(false);
+ }
+ return result;
+ }
+
+ std::string generate(const std::string bucket = "") const
+ {
+ std::string result = file_pattern;
+ if (need_replaced[0])
+ result = pattern_format(placeholders[0], toString(DB::UUIDHelpers::generateV4()));
+ if (need_replaced[1])
+ result = pattern_format(placeholders[1], bucket);
+ return result;
+ }
- std::string generate() const
+ std::string pattern_format(std::string arg, std::string replacement) const
Review Comment:
参数使用常量引用，否则这里会有内存拷贝
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2511163755
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen merged PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2511142847
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2500047565
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2502719048
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#discussion_r1866891726
##########
cpp-ch/local-engine/Parser/RelParsers/WriteRelParser.cpp:
##########
@@ -52,14 +52,15 @@ DB::ProcessorPtr make_sink(
const DB::Block & input_header,
const DB::Block & output_header,
const std::string & base_path,
- const FileNameGenerator & generator,
+ FileNameGenerator & generator,
Review Comment:
const
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2503539170
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2511469096
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#discussion_r1866910322
##########
cpp-ch/local-engine/Storages/Output/NormalFileWriter.h:
##########
@@ -230,20 +230,57 @@ class WriteStats : public WriteStatsBase
struct FileNameGenerator
{
- const bool pattern;
- const std::string filename_or_pattern;
+ // Align with org.apache.spark.sql.execution.FileNamePlaceHolder
+ const std::vector<std::string> placeholders = {"{id}", "{bucket}"};
+ // Align with placeholders above
+ const std::vector<bool> need_replaced;
+ const std::string file_pattern;
+
+ FileNameGenerator(const std::string& file_pattern)
+ : file_pattern(file_pattern), need_replaced(compute_need_replaced(file_pattern))
+ {
+ }
+
+ std::vector<bool> compute_need_replaced(const std::string & file_pattern)
+ {
+ std::vector<bool> result;
+ for(const std::string& placeholder: placeholders)
+ {
+ if (file_pattern.find(placeholder) != std::string::npos)
+ result.push_back(true);
+ else
+ result.push_back(false);
+ }
+ return result;
+ }
+
+ std::string generate(const std::string bucket = "") const
+ {
+ std::string result = file_pattern;
+ if (need_replaced[0])
+ result = pattern_format(placeholders[0], toString(DB::UUIDHelpers::generateV4()));
+ if (need_replaced[1])
+ result = pattern_format(placeholders[1], bucket);
+ return result;
+ }
- std::string generate() const
+ std::string pattern_format(std::string arg, std::string replacement) const
Review Comment:
参数使用引用
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#discussion_r1866903823
##########
cpp-ch/local-engine/Storages/Output/NormalFileWriter.h:
##########
@@ -230,20 +230,57 @@ class WriteStats : public WriteStatsBase
struct FileNameGenerator
{
- const bool pattern;
- const std::string filename_or_pattern;
+ // Align with org.apache.spark.sql.execution.FileNamePlaceHolder
+ const std::vector<std::string> placeholders = {"{id}", "{bucket}"};
Review Comment:
静态变量
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#issuecomment-2513451486
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7028][CH][Part-11] Support write parquet files with bucket [incubator-gluten]
Posted by "lwz9103 (via GitHub)" <gi...@apache.org>.
lwz9103 commented on code in PR #8052:
URL: https://github.com/apache/incubator-gluten/pull/8052#discussion_r1867178648
##########
cpp-ch/local-engine/Parser/RelParsers/WriteRelParser.cpp:
##########
@@ -52,14 +52,15 @@ DB::ProcessorPtr make_sink(
const DB::Block & input_header,
const DB::Block & output_header,
const std::string & base_path,
- const FileNameGenerator & generator,
+ FileNameGenerator & generator,
Review Comment:
resolved
##########
cpp-ch/local-engine/Storages/Output/NormalFileWriter.h:
##########
@@ -230,20 +230,57 @@ class WriteStats : public WriteStatsBase
struct FileNameGenerator
{
- const bool pattern;
- const std::string filename_or_pattern;
+ // Align with org.apache.spark.sql.execution.FileNamePlaceHolder
+ const std::vector<std::string> placeholders = {"{id}", "{bucket}"};
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org