You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "ZhangHuiGui (via GitHub)" <gi...@apache.org> on 2025/04/25 07:07:09 UTC
[I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
ZhangHuiGui opened a new issue, #2201:
URL: https://github.com/apache/orc/issues/2201
## Reproduce codes and orc-data
orc-version: current main, commit-id `3251a01f56d3f813e4889eb7d75eb2d0d63551f6`
Reproduced codes:
```c
#include <cmath>
#include <iostream>
#include <queue>
#include <list>
#include <orc/OrcFile.hh>
#include <orc/Reader.hh>
#include <orc/Type.hh>
#include <orc/Vector.hh>
#include <string>
#include <vector>
#define BATCH_SIZE 1024
void print_batch(const ORC_UNIQUE_PTR<orc::ColumnVectorBatch>& curr_batch,
const char* msg) {
std::cout << "debug check vector str " << msg << " curr_batch "
<< curr_batch.get() << std::endl;
auto curr_batch_data =
dynamic_cast<orc::StructVectorBatch*>(curr_batch.get());
auto string_batch =
dynamic_cast<orc::StringVectorBatch*>(curr_batch_data->fields[0]);
int64_t total = curr_batch->numElements;
for (int64_t idx = 0; idx < total; idx++) {
int64_t orc_data_len = string_batch->length[idx];
char* orc_data = (char*)string_batch->data[idx];
std::cout << "debug check vector str " << msg << " "
<< std::string(orc_data, orc_data_len) << std::endl;
}
}
int main(int argc, char* argv[]) {
if (argc < 2) {
std::cerr << " invalid args " << std::endl;
return 1;
}
int num_count = std::stoi(argv[1]);
try {
const std::string& filePath = "000000_0";
// reader
auto reader =
orc::createReader(orc::readFile(filePath), orc::ReaderOptions());
auto row_count = reader->getNumberOfRows();
// row_reader
orc::RowReaderOptions rowReaderOptions;
std::list<uint64_t> l;
l.push_back(6);
rowReaderOptions.include(l);
std::unique_ptr<orc::RowReader> rowReader =
reader->createRowReader(rowReaderOptions);
const orc::Type& schema = rowReader->getSelectedType();
// read colum batch to vector
std::vector<ORC_UNIQUE_PTR<orc::ColumnVectorBatch>> batch_pool;
std::unique_ptr<orc::ColumnVectorBatch> batch =
rowReader->createRowBatch(BATCH_SIZE);
batch_pool.reserve(row_count / BATCH_SIZE + 1);
int batch_count = 0;
while (rowReader->next(*batch) && batch_count < num_count) {
batch_pool.push_back(std::move(batch));
if (batch_pool.size() == 441 || batch_pool.size() == 1) print_batch(batch_pool[0], "inner loop");
batch = rowReader->createRowBatch(BATCH_SIZE);
batch_count++;
}
// BUG access the first batch in vector, all the data is invalid
print_batch(batch_pool[0], "after loop");
} catch (const std::exception& e) {
std::cerr << "Error: " << e.what() << std::endl;
return 1;
}
return 0;
}
```
Execute program with attached orc-data:
`./test_orc 440`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2832901711
Yes, the recommended approach is to reuse a single `ColumnVectorBatch` and consume it before calling `next`. The lifecycle is bound to the column reader. Maybe we should better document this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2833069282
Right, we don't have that yet. It is recommended to reuse the batch and consume it right away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2829876656
I suspect that `c_mktsegment` column has some nulls in it. You might need to check `hasNulls` and `notNull[i]` to confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "ZhangHuiGui (via GitHub)" <gi...@apache.org>.
ZhangHuiGui commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2832946416
Ok, I see.
So we don't have the clone ColumnVectorBatch interface yet, right? I don't see the header file containing the relevant interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2832916571
This is a common practice that the lifecycle of the reusable batch is bound to the last reader state. The reason is that creating a batch is usually a heavy operation due to memory allocation. This technique is also applied to `arrow-java` to reuse the `VectorSchemaRoot` object. It is just a coincidence that other kinds of batches do not have the lifecycle issue because only dictionary-encoded data can have the view semantics. To keep the data to live longer, it is better to provide a clone method to `ColumnVectorBatch`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "ZhangHuiGui (via GitHub)" <gi...@apache.org>.
ZhangHuiGui commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2832908096
> Yes, the recommended approach is to reuse a single `ColumnVectorBatch` and consume it before calling `next`. The lifecycle is bound to the column reader.
Thanks.
But I think this is not reasonable.
1. From the semantics of the Reader interface:
When the caller uses `createRowBatch` to create a new `ColumnVectorBatch`, and then executes Next to fill the data, the semantics should be that the current `ColumnVectorBatch` manages the life cycle of the data itself.
2. From the perspective of ColumnReader implementation, only the `StringDictionaryColumnReader` has this problem, and other readers do not have this limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "ZhangHuiGui (via GitHub)" <gi...@apache.org>.
ZhangHuiGui commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2830243816
I think there is a bug in StringDictionaryColumnReader which is the `c_mktsegment`'s column reader type in the example.
The ColumnVectorBatch's data fetched from `StringDictionaryColumnReader::next` is hold by `dictionary_`, when we switch stripe and reset the ColumnReader, the `dictionary_` and it's data will be reset either.
<img width="836" alt="Image" src="https://github.com/user-attachments/assets/81796883-5aee-444f-969e-0b1ec0893022" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "ZhangHuiGui (via GitHub)" <gi...@apache.org>.
ZhangHuiGui commented on issue #2201:
URL: https://github.com/apache/orc/issues/2201#issuecomment-2829908906
I have checked, the `c_mktsegment` column data has no null.
```shell
./orc_test 440 |grep hasNull
debug hasNull 0
debug hasNull 0
```
Could this be related to the reset of columnReader?
I saw that the reader was reset when reading the 440th batch.
<img width="837" alt="Image" src="https://github.com/user-attachments/assets/31ef46ae-f6a9-49a0-9e04-32b07038fedf" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next [orc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2201: Invalid ColumnVectorBatch access when fetch from std::vector which construct from a loop with RowReader::next
URL: https://github.com/apache/orc/issues/2201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org