You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jmckenna (via GitHub)" <gi...@apache.org> on 2024/08/29 15:06:00 UTC
[I] question: can this library be used directly by GDAL? [arrow-nanoarrow]
jmckenna opened a new issue, #600:
URL: https://github.com/apache/arrow-nanoarrow/issues/600
- background: see [Arrow requirements](https://gdal.org/en/latest/development/building_from_source.html#arrow) for compiling GDAL
- description:
- it would be beneficial for packagers of GDAL to be able to use the lightweight nanoarrow lib, to read GeoParquet files etc.
- but I am assuming that since the GDAL drivers for GeoParquet etc. allow creation, that nanoarrow cannot be used as it is for only read access
Are my assumptions correct?
Anyone here successfully compiled GDAL using nanoarrow instead of Arrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] question: can this library be used directly by GDAL to compile the GeoParquet driver? [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #600:
URL: https://github.com/apache/arrow-nanoarrow/issues/600#issuecomment-2319758923
Unfortunately, nanoarrow does not provide a Parquet implementation! It could in theory be used to support a Parquet implementation; however, that is definitely outside the scope of what we do here.
There is an "Arrow" driver; however, and we do have an IPC implementation that will (soon) be able to read and write. That would at least allow the Arrow driver to be built in by default!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] question: can this library be used directly by GDAL to compile the GeoParquet driver? [arrow-nanoarrow]
Posted by "jmckenna (via GitHub)" <gi...@apache.org>.
jmckenna commented on issue #600:
URL: https://github.com/apache/arrow-nanoarrow/issues/600#issuecomment-2320948142
To understand what I am asking: try to compile GDAL yourself, and in your CMake command enable Arrow support through `-DGDAL_USE_ARROW=ON` and point to nanoarrow, instead of Arrow. Then I think you will understand my initial question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] question: can this library be used directly by GDAL to compile the GeoParquet driver? [arrow-nanoarrow]
Posted by "jmckenna (via GitHub)" <gi...@apache.org>.
jmckenna commented on issue #600:
URL: https://github.com/apache/arrow-nanoarrow/issues/600#issuecomment-2320932745
Thanks for your explanation @paleolimbot. Closing.
I will try to explain what I was trying to ask, in case another GDAL packager comes back here a year from now, wondering the same.
Maybe I should have removed the mention of GeoParquet.
As you know, GDAL provides a GeoParquet driver, great. I would like to build that driver through GDAL. However the GeoParquet driver in GDAL requires the Arrow C++ API ; I came here and opened this ticket, to ask if I can instead use the nanoarrow library (I built it through CMake) instead of the full Arrow library, in my GDAL CMake command, to compile GDAL (with GeoParquet support).
I could see how people would ask you for direct Parquet implementation through nanoarrow, but, that wasn't what I was asking here.
I hope I am making a little more sense now, ha.
thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] question: can this library be used directly by GDAL to compile the GeoParquet driver? [arrow-nanoarrow]
Posted by "jmckenna (via GitHub)" <gi...@apache.org>.
jmckenna closed issue #600: question: can this library be used directly by GDAL to compile the GeoParquet driver?
URL: https://github.com/apache/arrow-nanoarrow/issues/600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] question: can this library be used directly by GDAL to compile the GeoParquet driver? [arrow-nanoarrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #600:
URL: https://github.com/apache/arrow-nanoarrow/issues/600#issuecomment-2322933380
To add one more potential clarification: Parquet is a file format, and so the Arrow C++ project just "happens" to have functionality to read and write that file format, similarly as it has also support for CSV, JSON, ORC, ... file formats (but Parquet is probably the main reason for people to use Arrow C++ for file IO).
On the other hand, the nanoarrow project, as a minimal library, has no support for reading/writing file formats (outside of Arrow IPC files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] question: can this library be used directly by GDAL to compile the GeoParquet driver? [arrow-nanoarrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #600:
URL: https://github.com/apache/arrow-nanoarrow/issues/600#issuecomment-2322930986
> GDAL provides a GeoParquet driver, great. I would like to build that driver through GDAL. However the GeoParquet driver in GDAL requires the Arrow C++ API ;
The GDAL Parquet driver does not just need the Arrow C++ _API_, but specifically the Arrow C++ library and its component to read and write Parquet files (i.e. `libparquet`, which is a separate component distributed by the Arrow C++ project, and which depends on `libarrow`).
So essentially what the GDAL driver needs is a Parquet implementation, not just an API to work with the Arrow in-memory format. So that's the reason that you can't replace Arrow C++ with nanoarrow, i.e. because Arrow C++ typically includes a Parquet implementation, while nanoarrow does not.
Does that make it clearer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] question: can this library be used directly by GDAL to compile the GeoParquet driver? [arrow-nanoarrow]
Posted by "jmckenna (via GitHub)" <gi...@apache.org>.
jmckenna commented on issue #600:
URL: https://github.com/apache/arrow-nanoarrow/issues/600#issuecomment-2324586745
@jorisvandenbossche thank-you for the detailed explanation, it makes sense to me now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org