You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/18 20:51:27 UTC
[GitHub] [arrow] visitor-4137 opened a new issue #10553: undefined reference to `LZ4F_resetDecompressionContext' error
visitor-4137 opened a new issue #10553:
URL: https://github.com/apache/arrow/issues/10553
I had written a cpp code to run on a rhel 8 machine which reads a csv file through a record batch reader and then writes these record batches to a feather file.. while compiling the program, i get the following error -
> /lib/../lib64/libarrow.so: undefined reference to `LZ4F_resetDecompressionContext'
The code i have is -
```
#include <arrow/api.h>
#include <arrow/filesystem/filesystem.h>
#include <arrow/filesystem/localfs.h>
#include <arrow/ipc/feather.h>
#include <arrow/ipc/writer.h>
#include <chrono>
#include <iostream>
#include <string>
#include <arrow/csv/api.h>
arrow::fs::LocalFileSystem file_system;
void make_table_from_csv(std::string file_name,std::string output_file) {
arrow::io::IOContext io_context = arrow::io::default_io_context();
// open the input file
std::shared_ptr<arrow::io::InputStream> input =
file_system.OpenInputStream(file_name).ValueOrDie();
std::shared_ptr<arrow::io::OutputStream> output =
file_system.OpenOutputStream(output_file).ValueOrDie();
auto read_options = arrow::csv::ReadOptions::Defaults();
read_options.block_size = (1 << 28); // decides what size of chunk we will read in a single record batch, increasing block_size increases size of each chunk.
auto parse_options = arrow::csv::ParseOptions::Defaults();
auto convert_options = arrow::csv::ConvertOptions::Defaults();
auto stream_reader = arrow::csv::StreamingReader::Make(io_context,input,read_options,parse_options,convert_options);
if (!stream_reader.ok()) {
std::cerr << "Error while reading csv file." << std::endl;
}
std::shared_ptr<arrow::csv::StreamingReader> reader = *stream_reader;
std::shared_ptr <arrow::RecordBatch> chunk;
reader -> ReadNext(&chunk);
auto schema = chunk -> schema();
auto rec_writer = arrow::ipc::MakeStreamWriter(output.get(),schema).ValueOrDie();
rec_writer -> WriteRecordBatch(*chunk);
std::unique_ptr <arrow::RecordBatchBuilder> rec_builder;
arrow::RecordBatchBuilder::Make(schema,arrow::default_memory_pool(),&rec_builder);
while(true){
rec_builder -> Flush(&chunk);
reader -> ReadNext(&chunk);
if(chunk == NULL){
break;
}
else{
rec_writer -> WriteRecordBatch(*chunk);
}
}
output -> Close();
return;
}
int main(int argc, char *argv[]) {
make_table_from_csv(std::string(argv[1]),"data.feather");
return 0;
}
```
Built the program with -
> g++ -O3 a.cpp -larrow
The LZ4 version is -
> *** LZ4 command line interface 64-bits v1.7.5, by Yann Collet ***
What is the error and why is it not working ?
I know updating the LZ4 could be a simple solution to this but as far as i know there is no such restriction of using a certain LZ4 version, correct me in case if i am wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #10553: undefined reference to `LZ4F_resetDecompressionContext' error
Posted by GitBox <gi...@apache.org>.
kou commented on issue #10553:
URL: https://github.com/apache/arrow/issues/10553#issuecomment-864849965
Our package for RHEL 8 is built with LZ4 1.8.0 or later. So our package uses a feature in LZ4 1.8.0 or later.
You need to use LZ4 1.8.0 or later with our package.
Our 4.0.0 package misses required LZ4 version information. But our 5.0.0 package requires LZ4 1.8.0 or later explicitly: https://github.com/apache/arrow/blob/master/dev/tasks/linux-packages/apache-arrow/yum/arrow.spec.in#L38-L41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #10553: undefined reference to `LZ4F_resetDecompressionContext' error
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #10553:
URL: https://github.com/apache/arrow/issues/10553#issuecomment-864839246
It seems `LZ4F_resetDecompressionContext` appeared in LZ4 1.8.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed issue #10553: undefined reference to `LZ4F_resetDecompressionContext' error
Posted by GitBox <gi...@apache.org>.
kou closed issue #10553:
URL: https://github.com/apache/arrow/issues/10553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #10553: undefined reference to `LZ4F_resetDecompressionContext' error
Posted by GitBox <gi...@apache.org>.
kou commented on issue #10553:
URL: https://github.com/apache/arrow/issues/10553#issuecomment-864849965
Our package for RHEL 8 is built with LZ4 1.8.0 or later. So our package uses a feature in LZ4 1.8.0 or later.
You need to use LZ4 1.8.0 or later with our package.
Our 4.0.0 package misses required LZ4 version information. But our 5.0.0 package requires LZ4 1.8.0 or later explicitly: https://github.com/apache/arrow/blob/master/dev/tasks/linux-packages/apache-arrow/yum/arrow.spec.in#L38-L41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #10553: undefined reference to `LZ4F_resetDecompressionContext' error
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #10553:
URL: https://github.com/apache/arrow/issues/10553#issuecomment-864839246
It seems `LZ4F_resetDecompressionContext` appeared in LZ4 1.8.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed issue #10553: undefined reference to `LZ4F_resetDecompressionContext' error
Posted by GitBox <gi...@apache.org>.
kou closed issue #10553:
URL: https://github.com/apache/arrow/issues/10553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org