You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/19 10:13:03 UTC
[GitHub] [doris] BePPPower opened a new pull request, #15175: [feature](new file reader) Integrate new file reader
BePPPower opened a new pull request, #15175:
URL: https://github.com/apache/doris/pull/15175
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
BePPPower commented on code in PR #15175:
URL: https://github.com/apache/doris/pull/15175#discussion_r1053018984
##########
be/src/runtime/routine_load/data_consumer_group.cpp:
##########
@@ -96,8 +96,8 @@ Status KafkaDataConsumerGroup::start_all(StreamLoadContext* ctx) {
int64_t left_rows = ctx->max_batch_rows;
int64_t left_bytes = ctx->max_batch_size;
- std::shared_ptr<KafkaConsumerPipe> kafka_pipe =
- std::static_pointer_cast<KafkaConsumerPipe>(ctx->body_sink);
+ std::shared_ptr<io::KafkaConsumerPipeReader> kafka_pipe =
Review Comment:
Maybe we can not use `KafkaConsumerPipe` because a class named `KafkaConsumerPipe` already exists.
##########
be/src/runtime/routine_load/data_consumer_group.cpp:
##########
@@ -96,8 +96,8 @@ Status KafkaDataConsumerGroup::start_all(StreamLoadContext* ctx) {
int64_t left_rows = ctx->max_batch_rows;
int64_t left_bytes = ctx->max_batch_size;
- std::shared_ptr<KafkaConsumerPipe> kafka_pipe =
- std::static_pointer_cast<KafkaConsumerPipe>(ctx->body_sink);
+ std::shared_ptr<io::KafkaConsumerPipeReader> kafka_pipe =
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #15175:
URL: https://github.com/apache/doris/pull/15175#discussion_r1053005459
##########
be/src/runtime/routine_load/data_consumer_group.cpp:
##########
@@ -96,8 +96,8 @@ Status KafkaDataConsumerGroup::start_all(StreamLoadContext* ctx) {
int64_t left_rows = ctx->max_batch_rows;
int64_t left_bytes = ctx->max_batch_size;
- std::shared_ptr<KafkaConsumerPipe> kafka_pipe =
- std::static_pointer_cast<KafkaConsumerPipe>(ctx->body_sink);
+ std::shared_ptr<io::KafkaConsumerPipeReader> kafka_pipe =
Review Comment:
```suggestion
std::shared_ptr<io::KafkaConsumerPipe> kafka_pipe =
```
##########
be/src/io/fs/stream_load_pipe.cpp:
##########
@@ -132,25 +138,26 @@ Status StreamLoadPipeReader::_read_next_buffer(std::unique_ptr<uint8_t[]>* data,
// finished
if (_buf_queue.empty()) {
DCHECK(_finished);
- data->reset();
- *length = 0;
+ // data->reset();
+ data = nullptr;
+ *bytes_read = 0;
return Status::OK();
}
auto buf = _buf_queue.front();
- *length = buf->remaining();
- data->reset(new uint8_t[*length]);
- buf->get_bytes((char*)(data->get()), *length);
+ *bytes_read = buf->remaining();
+ data = new char[*bytes_read];
Review Comment:
Memory leak?
##########
be/src/util/hdfs_storage_backend.cpp:
##########
@@ -18,7 +18,6 @@
#include "util/hdfs_storage_backend.h"
Review Comment:
I think this class can be replaced by `hdfs_filesystem`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15175:
URL: https://github.com/apache/doris/pull/15175#discussion_r1055149588
##########
be/src/vec/exec/vjson_scanner.cpp:
##########
@@ -140,8 +140,8 @@
if (_cur_vjson_reader != nullptr) {
_cur_vjson_reader.reset();
}
- std::string json_root;
- std::string jsonpath;
+ std::string json_root = "";
+ std::string jsonpath = "";
Review Comment:
warning: redundant string initialization [readability-redundant-string-init]
```suggestion
std::string jsonpath;
```
##########
be/src/vec/exec/vjson_scanner.cpp:
##########
@@ -140,8 +140,8 @@ Status VJsonScanner<JsonReader>::_open_vjson_reader() {
if (_cur_vjson_reader != nullptr) {
_cur_vjson_reader.reset();
}
- std::string json_root;
- std::string jsonpath;
+ std::string json_root = "";
Review Comment:
warning: redundant string initialization [readability-redundant-string-init]
```suggestion
std::string json_root;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15175:
URL: https://github.com/apache/doris/pull/15175#issuecomment-1364772578
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15175:
URL: https://github.com/apache/doris/pull/15175#issuecomment-1364772593
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15175:
URL: https://github.com/apache/doris/pull/15175#discussion_r1052044105
##########
be/src/vec/exec/format/parquet/parquet_thrift_util.h:
##########
@@ -33,12 +34,14 @@ namespace doris::vectorized {
constexpr uint8_t PARQUET_VERSION_NUMBER[4] = {'P', 'A', 'R', '1'};
constexpr uint32_t PARQUET_FOOTER_SIZE = 8;
-static Status parse_thrift_footer(FileReader* file, std::shared_ptr<FileMetaData>& file_metadata) {
+static Status parse_thrift_footer(io::FileReaderSPtr file,
Review Comment:
warning: unused function 'parse_thrift_footer' [clang-diagnostic-unused-function]
```cpp
static Status parse_thrift_footer(io::FileReaderSPtr file,
^
```
##########
be/test/exec/plain_text_line_reader_bzip_test.cpp:
##########
@@ -37,16 +37,17 @@ class PlainTextLineReaderTest : public testing::Test {
};
TEST_F(PlainTextLineReaderTest, bzip2_normal_use) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/test_file.csv.bz2",
- 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file(
+ "./be/test/exec/test_data/plain_text_line_reader/test_file.csv.bz2", &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::BZIP2, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_bzip_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_bzip_test.cpp:
##########
@@ -89,15 +90,17 @@
}
TEST_F(PlainTextLineReaderTest, bzip2_test_limit) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::BZIP2, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 8, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_bzip_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_bzip_test.cpp:
##########
@@ -150,15 +155,17 @@
}
TEST_F(PlainTextLineReaderTest, bzip2_test_limit3) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::BZIP2, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_bzip_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_bzip_test.cpp:
##########
@@ -180,15 +187,17 @@
}
TEST_F(PlainTextLineReaderTest, bzip2_test_limit4) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::BZIP2, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_bzip_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_gzip_test.cpp:
##########
@@ -186,15 +194,17 @@
}
TEST_F(PlainTextLineReaderGzipTest, gzip_test_limit3) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::GZIP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderGzipTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_gzip_test.cpp:37:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_gzip_test.cpp:
##########
@@ -37,16 +39,17 @@ class PlainTextLineReaderGzipTest : public testing::Test {
};
TEST_F(PlainTextLineReaderGzipTest, gzip_normal_use) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/test_file.csv.gz",
- 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file(
+ "./be/test/exec/test_data/plain_text_line_reader/test_file.csv.gz", &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::GZIP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderGzipTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_gzip_test.cpp:37:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_gzip_test.cpp:
##########
@@ -161,15 +167,17 @@
}
TEST_F(PlainTextLineReaderGzipTest, gzip_test_limit2) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::GZIP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 6, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderGzipTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_gzip_test.cpp:37:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_gzip_test.cpp:
##########
@@ -125,15 +129,17 @@
}
TEST_F(PlainTextLineReaderGzipTest, gzip_test_limit) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::GZIP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 8, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderGzipTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_gzip_test.cpp:37:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lz4frame_test.cpp:
##########
@@ -210,15 +218,17 @@
}
TEST_F(PlainTextLineReaderTest, lz4_test_limit5) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZ4FRAME, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 0, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lz4frame_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_gzip_test.cpp:
##########
@@ -216,15 +226,17 @@
}
TEST_F(PlainTextLineReaderGzipTest, gzip_test_limit4) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::GZIP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderGzipTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_gzip_test.cpp:37:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lz4frame_test.cpp:
##########
@@ -150,15 +154,17 @@
}
TEST_F(PlainTextLineReaderTest, lz4_test_limit3) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZ4FRAME, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lz4frame_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lzop_test.cpp:
##########
@@ -176,15 +182,17 @@
}
TEST_F(PlainTextLineReaderTest, lzop_test_limit4) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZOP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lzop_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lz4frame_test.cpp:
##########
@@ -126,15 +128,17 @@
}
TEST_F(PlainTextLineReaderTest, lz4_test_limit2) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZ4FRAME, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 6, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lz4frame_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_bzip_test.cpp:
##########
@@ -126,15 +129,17 @@
}
TEST_F(PlainTextLineReaderTest, bzip2_test_limit2) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::BZIP2, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 6, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_bzip_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_bzip_test.cpp:
##########
@@ -210,15 +219,17 @@
}
TEST_F(PlainTextLineReaderTest, bzip2_test_limit5) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.bz2",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::BZIP2, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 0, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_bzip_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lz4frame_test.cpp:
##########
@@ -37,16 +36,17 @@ class PlainTextLineReaderTest : public testing::Test {
};
TEST_F(PlainTextLineReaderTest, lz4_normal_use) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/test_file.csv.lz4",
- 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file(
+ "./be/test/exec/test_data/plain_text_line_reader/test_file.csv.lz4", &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZ4FRAME, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lz4frame_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lzop_test.cpp:
##########
@@ -124,15 +126,17 @@
}
TEST_F(PlainTextLineReaderTest, lzop_test_limit2) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZOP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 6, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lzop_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_gzip_test.cpp:
##########
@@ -246,15 +258,17 @@
}
TEST_F(PlainTextLineReaderGzipTest, gzip_test_limit5) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.gz",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::GZIP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 0, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderGzipTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_gzip_test.cpp:37:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_gzip_test.cpp:
##########
@@ -89,16 +92,17 @@
}
TEST_F(PlainTextLineReaderGzipTest, uncompressed_no_newline) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/no_newline.csv.gz",
- 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file(
+ "./be/test/exec/test_data/plain_text_line_reader/no_newline.csv.gz", &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::GZIP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderGzipTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_gzip_test.cpp:37:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lzop_test.cpp:
##########
@@ -88,15 +88,17 @@
}
TEST_F(PlainTextLineReaderTest, lzop_test_limit) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZOP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 8, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lzop_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lzop_test.cpp:
##########
@@ -37,16 +36,17 @@ class PlainTextLineReaderTest : public testing::Test {
};
TEST_F(PlainTextLineReaderTest, lzop_normal_use) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/test_file.csv.lzo",
- 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file(
+ "./be/test/exec/test_data/plain_text_line_reader/test_file.csv.lzo", &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZOP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lzop_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lzop_test.cpp:
##########
@@ -147,15 +151,17 @@
}
TEST_F(PlainTextLineReaderTest, lzop_test_limit3) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZOP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lzop_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lz4frame_test.cpp:
##########
@@ -89,15 +89,17 @@
}
TEST_F(PlainTextLineReaderTest, lz4_test_limit) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZ4FRAME, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 8, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lz4frame_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_lz4frame_test.cpp:
##########
@@ -180,15 +186,17 @@
}
TEST_F(PlainTextLineReaderTest, lz4_test_limit4) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lz4",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZ4FRAME, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lz4frame_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15175:
URL: https://github.com/apache/doris/pull/15175#discussion_r1052399910
##########
be/test/exec/plain_text_line_reader_lzop_test.cpp:
##########
@@ -205,15 +213,17 @@ TEST_F(PlainTextLineReaderTest, lzop_test_limit4) {
}
TEST_F(PlainTextLineReaderTest, lzop_test_limit5) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv.lzo",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZOP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 0, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lzop_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -88,17 +90,18 @@
}
TEST_F(PlainTextLineReaderUncompressedTest, uncompressed_no_newline) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/no_newline.csv",
- 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/no_newline.csv",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::UNCOMPRESSED, &decompressor);
EXPECT_TRUE(st.ok());
EXPECT_TRUE(decompressor == nullptr);
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -124,16 +127,18 @@
}
TEST_F(PlainTextLineReaderUncompressedTest, uncompressed_test_limit) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::UNCOMPRESSED, &decompressor);
EXPECT_TRUE(st.ok());
EXPECT_TRUE(decompressor == nullptr);
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 8, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 8, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -37,16 +37,18 @@ class PlainTextLineReaderUncompressedTest : public testing::Test {
};
TEST_F(PlainTextLineReaderUncompressedTest, uncompressed_normal_use) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/test_file.csv", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/test_file.csv",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::UNCOMPRESSED, &decompressor);
EXPECT_TRUE(st.ok());
EXPECT_TRUE(decompressor == nullptr);
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -280,7 +295,7 @@
EXPECT_TRUE(decompressor == nullptr);
// set min length larger than 0 to test
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 10, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 10, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 10, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -218,16 +227,18 @@
}
TEST_F(PlainTextLineReaderUncompressedTest, uncompressed_test_limit4) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::UNCOMPRESSED, &decompressor);
EXPECT_TRUE(st.ok());
EXPECT_TRUE(decompressor == nullptr);
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -249,16 +260,18 @@
}
TEST_F(PlainTextLineReaderUncompressedTest, uncompressed_test_limit5) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::UNCOMPRESSED, &decompressor);
EXPECT_TRUE(st.ok());
EXPECT_TRUE(decompressor == nullptr);
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 0, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 0, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -52,7 +51,6 @@
_env = ExecEnv::GetInstance();
_env->_thread_mgr = new ThreadResourceMgr();
_env->_master_info = new TMasterInfo();
- _env->_load_stream_mgr = new LoadStreamMgr();
_env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
_env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
Review Comment:
warning: '_function_client_cache' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env->_function_client_cache = new BrpcClientCache<PFunctionService_Stub>();
^
```
**be/src/runtime/exec_env.h:256:** declared private here
```cpp
BrpcClientCache<PFunctionService_Stub>* _function_client_cache = nullptr;
^
```
##########
be/test/runtime/routine_load_task_executor_test.cpp:
##########
@@ -58,8 +58,8 @@
void TearDown() override {
delete _env._master_info;
_env._master_info = nullptr;
Review Comment:
warning: '_master_info' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env._master_info = nullptr;
^
```
**be/src/runtime/exec_env.h:245:** declared private here
```cpp
TMasterInfo* _master_info = nullptr;
^
```
##########
be/test/exec/plain_text_line_reader_lzop_test.cpp:
##########
@@ -224,16 +234,17 @@
}
TEST_F(PlainTextLineReaderTest, lzop_test_larger) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/larger.txt.lzo",
- 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/larger.txt.lzo",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::LZOP, &decompressor);
EXPECT_TRUE(st.ok());
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, -1, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, -1, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_lzop_test.cpp:34:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/runtime/routine_load_task_executor_test.cpp:
##########
@@ -58,8 +58,8 @@
void TearDown() override {
delete _env._master_info;
Review Comment:
warning: '_master_info' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
delete _env._master_info;
^
```
**be/src/runtime/exec_env.h:245:** declared private here
```cpp
TMasterInfo* _master_info = nullptr;
^
```
##########
be/test/runtime/routine_load_task_executor_test.cpp:
##########
@@ -48,7 +48,7 @@
k_stream_load_put_result = TStreamLoadPutResult();
_env._master_info = new TMasterInfo();
- _env._load_stream_mgr = new LoadStreamMgr();
+ _env._new_load_stream_mgr = new NewLoadStreamMgr();
_env._stream_load_executor = new StreamLoadExecutor(&_env);
Review Comment:
warning: '_stream_load_executor' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env._stream_load_executor = new StreamLoadExecutor(&_env);
^
```
**be/src/runtime/exec_env.h:262:** declared private here
```cpp
StreamLoadExecutor* _stream_load_executor = nullptr;
^
```
##########
be/test/runtime/routine_load_task_executor_test.cpp:
##########
@@ -58,8 +58,8 @@
void TearDown() override {
delete _env._master_info;
_env._master_info = nullptr;
- delete _env._load_stream_mgr;
- _env._load_stream_mgr = nullptr;
+ delete _env._new_load_stream_mgr;
+ _env._new_load_stream_mgr = nullptr;
Review Comment:
warning: '_new_load_stream_mgr' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env._new_load_stream_mgr = nullptr;
^
```
**be/src/runtime/exec_env.h:254:** declared private here
```cpp
NewLoadStreamMgr* _new_load_stream_mgr = nullptr;
^
```
##########
be/test/runtime/routine_load_task_executor_test.cpp:
##########
@@ -58,8 +58,8 @@
void TearDown() override {
delete _env._master_info;
_env._master_info = nullptr;
- delete _env._load_stream_mgr;
- _env._load_stream_mgr = nullptr;
+ delete _env._new_load_stream_mgr;
Review Comment:
warning: '_new_load_stream_mgr' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
delete _env._new_load_stream_mgr;
^
```
**be/src/runtime/exec_env.h:254:** declared private here
```cpp
NewLoadStreamMgr* _new_load_stream_mgr = nullptr;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -161,16 +166,18 @@
}
TEST_F(PlainTextLineReaderUncompressedTest, uncompressed_test_limit2) {
- LocalFileReader file_reader("./be/test/exec/test_data/plain_text_line_reader/limit.csv", 0);
- auto st = file_reader.open();
+ io::FileSystemSPtr local_fs = std::make_shared<io::LocalFileSystem>("");
+ io::FileReaderSPtr file_reader;
+ auto st = local_fs->open_file("./be/test/exec/test_data/plain_text_line_reader/limit.csv",
+ &file_reader);
EXPECT_TRUE(st.ok());
Decompressor* decompressor;
st = Decompressor::create_decompressor(CompressType::UNCOMPRESSED, &decompressor);
EXPECT_TRUE(st.ok());
EXPECT_TRUE(decompressor == nullptr);
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 6, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 6, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/plain_text_line_reader_uncompressed_test.cpp:
##########
@@ -196,7 +205,7 @@
EXPECT_TRUE(st.ok());
EXPECT_TRUE(decompressor == nullptr);
- PlainTextLineReader line_reader(&_profile, &file_reader, decompressor, 7, "\n", 1);
+ NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
Review Comment:
warning: '_profile' is a private member of 'doris::PlainTextLineReaderUncompressedTest' [clang-diagnostic-error]
```cpp
NewPlainTextLineReader line_reader(&_profile, file_reader, decompressor, 7, "\n", 1, 0);
^
```
**be/test/exec/plain_text_line_reader_uncompressed_test.cpp:35:** declared private here
```cpp
RuntimeProfile _profile;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -52,7 +51,6 @@
_env = ExecEnv::GetInstance();
_env->_thread_mgr = new ThreadResourceMgr();
_env->_master_info = new TMasterInfo();
Review Comment:
warning: '_master_info' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env->_master_info = new TMasterInfo();
^
```
**be/src/runtime/exec_env.h:245:** declared private here
```cpp
TMasterInfo* _master_info = nullptr;
^
```
##########
be/test/runtime/routine_load_task_executor_test.cpp:
##########
@@ -48,7 +48,7 @@ class RoutineLoadTaskExecutorTest : public testing::Test {
k_stream_load_put_result = TStreamLoadPutResult();
_env._master_info = new TMasterInfo();
Review Comment:
warning: '_master_info' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env._master_info = new TMasterInfo();
^
```
**be/src/runtime/exec_env.h:245:** declared private here
```cpp
TMasterInfo* _master_info = nullptr;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -67,7 +65,6 @@
void TearDown() override {
SAFE_DELETE(_env->_internal_client_cache);
SAFE_DELETE(_env->_function_client_cache);
Review Comment:
warning: '_function_client_cache' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
SAFE_DELETE(_env->_function_client_cache);
^
```
**be/src/runtime/exec_env.h:256:** declared private here
```cpp
BrpcClientCache<PFunctionService_Stub>* _function_client_cache = nullptr;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -67,7 +65,6 @@
void TearDown() override {
SAFE_DELETE(_env->_internal_client_cache);
Review Comment:
warning: '_internal_client_cache' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
SAFE_DELETE(_env->_internal_client_cache);
^
```
**be/src/runtime/exec_env.h:255:** declared private here
```cpp
BrpcClientCache<PBackendService_Stub>* _internal_client_cache = nullptr;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -52,7 +51,6 @@ class OlapTableSinkTest : public testing::Test {
_env = ExecEnv::GetInstance();
_env->_thread_mgr = new ThreadResourceMgr();
Review Comment:
warning: '_thread_mgr' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env->_thread_mgr = new ThreadResourceMgr();
^
```
**be/src/runtime/exec_env.h:209:** declared private here
```cpp
ThreadResourceMgr* _thread_mgr = nullptr;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -67,7 +65,6 @@
void TearDown() override {
SAFE_DELETE(_env->_internal_client_cache);
SAFE_DELETE(_env->_function_client_cache);
- SAFE_DELETE(_env->_load_stream_mgr);
SAFE_DELETE(_env->_master_info);
Review Comment:
warning: '_master_info' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
SAFE_DELETE(_env->_master_info);
^
```
**be/src/runtime/exec_env.h:245:** declared private here
```cpp
TMasterInfo* _master_info = nullptr;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -52,7 +51,6 @@
_env = ExecEnv::GetInstance();
_env->_thread_mgr = new ThreadResourceMgr();
_env->_master_info = new TMasterInfo();
- _env->_load_stream_mgr = new LoadStreamMgr();
_env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
Review Comment:
warning: '_internal_client_cache' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env->_internal_client_cache = new BrpcClientCache<PBackendService_Stub>();
^
```
**be/src/runtime/exec_env.h:255:** declared private here
```cpp
BrpcClientCache<PBackendService_Stub>* _internal_client_cache = nullptr;
^
```
##########
be/test/exec/tablet_sink_test.cpp:
##########
@@ -67,7 +65,6 @@
void TearDown() override {
SAFE_DELETE(_env->_internal_client_cache);
SAFE_DELETE(_env->_function_client_cache);
- SAFE_DELETE(_env->_load_stream_mgr);
SAFE_DELETE(_env->_master_info);
SAFE_DELETE(_env->_thread_mgr);
Review Comment:
warning: '_thread_mgr' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
SAFE_DELETE(_env->_thread_mgr);
^
```
**be/src/runtime/exec_env.h:209:** declared private here
```cpp
ThreadResourceMgr* _thread_mgr = nullptr;
^
```
##########
be/test/runtime/routine_load_task_executor_test.cpp:
##########
@@ -48,7 +48,7 @@
k_stream_load_put_result = TStreamLoadPutResult();
_env._master_info = new TMasterInfo();
- _env._load_stream_mgr = new LoadStreamMgr();
+ _env._new_load_stream_mgr = new NewLoadStreamMgr();
Review Comment:
warning: '_new_load_stream_mgr' is a private member of 'doris::ExecEnv' [clang-diagnostic-error]
```cpp
_env._new_load_stream_mgr = new NewLoadStreamMgr();
^
```
**be/src/runtime/exec_env.h:254:** declared private here
```cpp
NewLoadStreamMgr* _new_load_stream_mgr = nullptr;
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #15175:
URL: https://github.com/apache/doris/pull/15175#issuecomment-1357461542
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.88 seconds
load time: 641 seconds
storage size: 17123100796 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221219105956_clickbench_pr_65100.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
BePPPower commented on code in PR #15175:
URL: https://github.com/apache/doris/pull/15175#discussion_r1053056071
##########
be/src/io/fs/stream_load_pipe.cpp:
##########
@@ -132,25 +138,26 @@ Status StreamLoadPipeReader::_read_next_buffer(std::unique_ptr<uint8_t[]>* data,
// finished
if (_buf_queue.empty()) {
DCHECK(_finished);
- data->reset();
- *length = 0;
+ // data->reset();
+ data = nullptr;
+ *bytes_read = 0;
return Status::OK();
}
auto buf = _buf_queue.front();
- *length = buf->remaining();
- data->reset(new uint8_t[*length]);
- buf->get_bytes((char*)(data->get()), *length);
+ *bytes_read = buf->remaining();
+ data = new char[*bytes_read];
Review Comment:
There are no leaks here because the data pointer is derived from the unique_ptr of the external call(in NewJsonReader::_read_one_message() ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
BePPPower commented on code in PR #15175:
URL: https://github.com/apache/doris/pull/15175#discussion_r1053018984
##########
be/src/runtime/routine_load/data_consumer_group.cpp:
##########
@@ -96,8 +96,8 @@ Status KafkaDataConsumerGroup::start_all(StreamLoadContext* ctx) {
int64_t left_rows = ctx->max_batch_rows;
int64_t left_bytes = ctx->max_batch_size;
- std::shared_ptr<KafkaConsumerPipe> kafka_pipe =
- std::static_pointer_cast<KafkaConsumerPipe>(ctx->body_sink);
+ std::shared_ptr<io::KafkaConsumerPipeReader> kafka_pipe =
Review Comment:
Maybe we can not use `KafkaConsumerPipe` because a class named `KafkaConsumerPipe` already exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #15175: [feature](new file reader) Integrate new file reader
Posted by GitBox <gi...@apache.org>.
morningman merged PR #15175:
URL: https://github.com/apache/doris/pull/15175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org