[GitHub] [arrow] jorisvandenbossche commented on pull request #34616: GH-29238 [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API

["jorisvandenbossche (via GitHub)" <gi...@apache.org>]

jorisvandenbossche commented on PR #34616:
URL: https://github.com/apache/arrow/pull/34616#issuecomment-1678630935

   It does _build_, but I think with the current changes to the dataset cython code, I would still expect that it will raise an error at run-time. 
   
   Testing that locally while fetching this branch to review it, I first build it with my normal setup that has parquet enabled but not encryption, and then my build indeed passes, but trying to use Parquet through the dataset API raises an error:
   
   ```
   In [1]: import pyarrow.dataset as ds
   
   In [2]: ds.dataset("test.parquet")
   ...
   
   File ~/scipy/repos/arrow/python/pyarrow/dataset.py:298, in _ensure_format(obj)
       296 elif obj == "parquet":
       297     if not _parquet_available:
   --> 298         raise ValueError(_parquet_msg)
       299     return ParquetFileFormat()
       300 elif obj in {"ipc", "arrow"}:
   
   ValueError: The pyarrow installation is not built with support for the Parquet file format.
   ```
   
   It thinks that Parquet is not available, because it cannot import then ``pyarrow._dataset_parquet`` cython module. And this is because that module now tries to import the encryption module, which isn't available in my installation:
   
   ```
   In [3]: import pyarrow._dataset_parquet
   ...
   ModuleNotFoundError: No module named 'pyarrow._parquet_encryption'
   ```
   
   To verify this (and also in general ensure this is covered), we should check with one of the CI builds that have parquet encryption disabled (although given all builds are green, we might lack such CI build).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org