You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/05 04:56:46 UTC
[GitHub] [arrow] ex791857 opened a new issue, #13801: [cpp][parquet] Segmentation fault when open parquet file
ex791857 opened a new issue, #13801:
URL: https://github.com/apache/arrow/issues/13801
Hi,
I am new to Arrow and Parquet.
I installed Arrow 9.0.0 and Parquet following the guide in this repo.
When I tried to open the parquet file using follow CPP code (same as the example code in docs), I got a segmentation fault showing below when calling `parquet::arrow::OpenFile`
```CPP
arrow::Status st;
arrow::MemoryPool *pool = arrow::default_memory_pool();
arrow::fs::LocalFileSystem file_system;
std::shared_ptr<arrow::io::RandomAccessFile> input =
file_system.OpenInputFile("/root/path/to/file.parquet").ValueOrDie();
// Open Parquet file reader
std::unique_ptr<parquet::arrow::FileReader> arrow_reader;
st = parquet::arrow::OpenFile(input, pool, &arrow_reader);
```
I backtracked the stack info in gdb:
```bash
(gdb) bt
#0 0x00007ffff5ed1f74 in arrow::Schema::num_fields() const () from /lib64/libarrow.so.800
#1 0x00007ffff7788a41 in parquet::arrow::SchemaManifest::Make(parquet::SchemaDescriptor const*, std::shared_ptr<arrow::KeyValueMetadata const> const&, parquet::ArrowReaderProperties const&, parquet::arrow::SchemaManifest*) () from /lib64/libparquet.so.800
#2 0x00007ffff7755395 in parquet::arrow::FileReader::Make(arrow::MemoryPool*, std::unique_ptr<parquet::ParquetFileReader, std::default_delete<parquet::ParquetFileReader> >, parquet::ArrowReaderProperties const&, std::unique_ptr<parquet::arrow::FileReader, std::default_delete<parquet::arrow::FileReader> >*) () from /lib64/libparquet.so.800
#3 0x00007ffff7755601 in parquet::arrow::FileReaderBuilder::Build(std::unique_ptr<parquet::arrow::FileReader, std::default_delete<parquet::arrow::FileReader> >*) () from /lib64/libparquet.so.800
#4 0x00007ffff77563b9 in parquet::arrow::OpenFile(std::shared_ptr<arrow::io::RandomAccessFile>, arrow::MemoryPool*, std::unique_ptr<parquet::arrow::FileReader, std::default_delete<parquet::arrow::FileReader> >*) () from /lib64/libparquet.so.800
#5 0x000000000040a4b6 in main (argc=1, argv=0x7fffffffdba8) at /root/mini-sim/mdl/src/main.cpp:65
(gdb)
```
But I can open and read the same file using pandas in python
```python
pandas.read_parquet("/root/path/to/file.parquet")
```
What did I miss?
I really appreciate any help you can provide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
westonpace commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1206628148
Is there any chance you could share your complete application and a sample parquet file so that we could run it and try it out for ourselves?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ex791857 commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
ex791857 commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1206046132
I also try this example. I got the same result as the above one, segmentation fault when calling `parquet::arrow::OpenFile`
```CPP
std::shared_ptr<arrow::io::ReadableFile> infile;
PARQUET_ASSIGN_OR_THROW(infile,
arrow::io::ReadableFile::Open("/root/path/to/file.parquet",
arrow::default_memory_pool()));
std::unique_ptr<parquet::arrow::FileReader> reader;
PARQUET_THROW_NOT_OK(parquet::arrow::OpenFile(infile, arrow::default_memory_pool(), &reader));
std::shared_ptr<arrow::Table> table;
PARQUET_THROW_NOT_OK(reader->ReadTable(&table));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
westonpace commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1209635633
Thank you so much for the detailed reproducer. Unfortunately, I have tried a few different variations (both just uncommenting the lines and actually running causeSegmenetationFault) and I have not been able to reproduce the issue. I am testing against the 9.0.0 release.
Do you know what version you are using?
Based on the strangeness of the error (e.g. triggered by code that is not executed) I wonder if the problem might be with memory corruption elsewhere or possibly by compiling and linking against different versions of the Arrow library.
Do you still get a segmentation fault when you run the example you shared?
Can you explain a bit more how you are compiling and linking Arrow (in particular, are you sure the header files you are compiling with come from the exact same version as the library you are linking with?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ex791857 commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
ex791857 commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1206160968
It turns out `std::make_shared<arrow::Schema>` is responsible for the segmentation fault. After I comment following code, the code could run normally.
```CPP
auto expected_schema = std::make_shared<arrow::Schema>(schema_vector);
if (!expected_schema->Equals(*table->schema()))
{
// The table doesn't have the expected schema thus we cannot directly
// convert it to our target representation.
spdlog::error("Schemas in {} are not matching order!", parFile);
exit(-1);
}
```
But I still got questions about how could a statement that never runs affect this library memory?
This is an example code from official docs https://arrow.apache.org/docs/cpp/examples/row_columnar_conversion.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ex791857 commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
ex791857 commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1206112423
I looked into Arrow source code and tracked to
```CPP
int Schema::num_fields() const { return static_cast<int>(impl_->fields_.size()); }
```
Maybe impl_ is a nullptr which causes a segmentation fault. But impl_ should be initialized when constructing the Schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ex791857 commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
ex791857 commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1210069242
I still get a segfault when I run the example. I compiled the example using CMakeList provided in the repo. There is only one version of arrow in my env.
Here is cmake find_package debug info
```bash
Checking file [/usr/lib64/cmake/arrow/ArrowConfig.cmake]
Checking file [/usr/lib64/cmake/arrow/ParquetConfig.cmake]
```
I check the version info in the above .cmake files, they are both 9.0.0
Here is my package info:
```bash
[root@3df06b313282 mdl]# yum list installed | grep arrow
apache-arrow-release.noarch 9.0.0-1.el7 installed
arrow-dataset-devel.x86_64 9.0.0-1.el7 @apache-arrow-centos
arrow-dataset-glib-devel.x86_64 9.0.0-1.el7 @apache-arrow-centos
arrow-devel.x86_64 9.0.0-1.el7 @apache-arrow-centos
arrow-glib-devel.x86_64 9.0.0-1.el7 @apache-arrow-centos
arrow9-dataset-glib-libs.x86_64 9.0.0-1.el7 @apache-arrow-centos
arrow9-dataset-libs.x86_64 9.0.0-1.el7 @apache-arrow-centos
arrow9-glib-libs.x86_64 9.0.0-1.el7 @apache-arrow-centos
arrow9-libs.x86_64 9.0.0-1.el7 @apache-arrow-centos
parquet-devel.x86_64 9.0.0-1.el7 @apache-arrow-centos
parquet-glib-devel.x86_64 9.0.0-1.el7 @apache-arrow-centos
parquet9-glib-libs.x86_64 9.0.0-1.el7 @apache-arrow-centos
parquet9-libs.x86_64 9.0.0-1.el7 @apache-arrow-centos
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
pitrou commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1215622263
@ex791857 Can you check that you don't have another Arrow install somewhere? `locate arrow/api.h` might help locate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ex791857 commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
ex791857 commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1207622818
Of course. I have extracted a minimum example to segmentation fault in this repo
https://github.com/ex791857/arrow-report
You can follow README to run this example.
Feel free to try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ex791857 commented on issue #13801: [cpp][parquet] Segmentation fault when open parquet file
Posted by GitBox <gi...@apache.org>.
ex791857 commented on issue #13801:
URL: https://github.com/apache/arrow/issues/13801#issuecomment-1217429875
@pitrou There is only one `arrow/api.h` in my env.
```bash
# locate arrow/api.h
/usr/include/arrow/api.h
```
I had pyarrow in my env, but after `pip uninstall pyarrow`, this bug still exists.
Maybe I can provide you with a docker later if I can reproduce this bug in it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org