You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "loneylee (via GitHub)" <gi...@apache.org> on 2024/08/14 09:23:23 UTC
[PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
loneylee opened a new pull request, #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
(Fixes: \#6840)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#discussion_r1722770664
##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -85,6 +85,16 @@ DB::ContextMutablePtr QueryContextManager::currentQueryContext()
return query_map.get(id)->query_context;
}
+std::shared_ptr<DB::ThreadGroup> QueryContextManager::currentThreadGroup()
Review Comment:
不需要这么复杂，直接返回currentThread::getGroup,不存在报错即可，这里会保证所有的线程都会attach到thread group
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2288454551
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2288273250
https://github.com/apache/incubator-gluten/issues/6840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#discussion_r1722599954
##########
cpp-ch/local-engine/Storages/SubstraitSource/ReadBufferBuilder.cpp:
##########
@@ -205,25 +202,31 @@ class LocalFileReadBufferBuilder : public ReadBufferBuilder
#if USE_HDFS
class HDFSFileReadBufferBuilder : public ReadBufferBuilder
{
+ using ReadBufferCreator = std::function<std::unique_ptr<DB::ReadBufferFromFileBase>(bool restricted_seek, const DB::StoredObject & object)>;
public:
explicit HDFSFileReadBufferBuilder(DB::ContextPtr context_) : ReadBufferBuilder(context_), context(context_) { }
~HDFSFileReadBufferBuilder() override = default;
std::unique_ptr<DB::ReadBuffer>
build(const substrait::ReadRel::LocalFiles::FileOrFiles & file_info, bool set_read_util_position) override
{
- auto config = HdfsConfig::loadFromContext(context);
+ DB::ReadSettings read_settings = getReadSettings(context);
+ auto & config = context->getConfigRef();
+ auto hdfs_config = HdfsConfig::loadFromContext(config, read_settings);
Poco::URI file_uri(file_info.uri_file());
std::string uri_path = "hdfs://" + file_uri.getHost();
if (file_uri.getPort())
- uri_path += ":" + std::to_string(file_uri.getPort());
+ uri_path += ":" + std::to_string(static_cast<unsigned>(file_uri.getPort()));
- DB::ReadSettings read_settings;
- std::unique_ptr<DB::ReadBuffer> read_buffer;
+ std::cout << "read_from_filesystem_cache_if_exists_otherwise_bypass_cache:" << read_settings.read_from_filesystem_cache_if_exists_otherwise_bypass_cache << std::endl;
Review Comment:
remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "loneylee (via GitHub)" <gi...@apache.org>.
loneylee commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2298513843
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "loneylee (via GitHub)" <gi...@apache.org>.
loneylee commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2300725443
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "loneylee (via GitHub)" <gi...@apache.org>.
loneylee merged PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2298336160
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "loneylee (via GitHub)" <gi...@apache.org>.
loneylee commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2297926889
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "loneylee (via GitHub)" <gi...@apache.org>.
loneylee commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2298121020
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2290826833
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2291026982
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "loneylee (via GitHub)" <gi...@apache.org>.
loneylee commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2300185617
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#discussion_r1722603009
##########
cpp-ch/local-engine/Parser/RelMetric.cpp:
##########
@@ -18,12 +18,57 @@
#include <Processors/IProcessor.h>
#include <Processors/QueryPlan/AggregatingStep.h>
#include <Processors/QueryPlan/ReadFromMergeTree.h>
+#include <Storages/SubstraitSource/SubstraitFileSourceStep.h>
using namespace rapidjson;
+namespace ProfileEvents
+{
+extern const Event FileSegmentWaitReadBufferMicroseconds;
+extern const Event FileSegmentReadMicroseconds;
+extern const Event FileSegmentCacheWriteMicroseconds;
+extern const Event FileSegmentPredownloadMicroseconds;
+extern const Event FileSegmentUsedBytes;
+
+extern const Event CachedReadBufferReadFromSourceMicroseconds;
+extern const Event CachedReadBufferReadFromCacheMicroseconds;
+extern const Event CachedReadBufferCacheWriteMicroseconds;
+extern const Event CachedReadBufferReadFromSourceBytes;
+extern const Event CachedReadBufferReadFromCacheBytes;
+extern const Event CachedReadBufferCacheWriteBytes;
+extern const Event CachedReadBufferCreateBufferMicroseconds;
+
+extern const Event CachedReadBufferReadFromCacheHits;
+extern const Event CachedReadBufferReadFromCacheMisses;
+}
+
namespace local_engine
{
+static void writeCacheHits(Writer<StringBuffer> & writer)
+{
+ auto & counters = DB::CurrentThread::getProfileEvents();
Review Comment:
用ThreadGroup获取，不要用currentThread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2298762099
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2290224149
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2290776109
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2288316170
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2288273696
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6840][CH] Enable cache files for hdfs [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6841:
URL: https://github.com/apache/incubator-gluten/pull/6841#issuecomment-2292944453
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org