You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2025/11/21 12:19:52 UTC
[PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
raulcd opened a new pull request, #48201:
URL: https://github.com/apache/arrow/pull/48201
### Rationale for this change
The Parquet C++ implementation contains information about some metadata that is not exposed to PyArrow. This PR is to expose those.
### What changes are included in this PR?
Expose present `parquet::ColumnChunkMetaData`:
- has_index_page
- index_page_offset
- bloom_filter_offset
- bloom_filter_length
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes, new API methods on pyarrow.parquet ColumnChunkMetadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #48201:
URL: https://github.com/apache/arrow/pull/48201#discussion_r2549592190
##########
python/pyarrow/tests/parquet/test_metadata.py:
##########
@@ -814,3 +816,31 @@ def msg(c):
with pytest.raises(TypeError, match=msg("FileMetaData")):
pq.FileMetaData()
+
+
+def test_column_metadata_with_bloom_filter(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'data_index_bloom_encoding_with_length.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is False
+ assert metadata.row_group(0).column(0).index_page_offset is None
+ assert metadata.row_group(0).column(0).has_bloom_filter is True
+ assert metadata.row_group(0).column(0).has_column_index is True
+ assert metadata.row_group(0).column(0).has_offset_index is True
+ assert metadata.row_group(0).column(0).bloom_filter_offset == 253
+ assert metadata.row_group(0).column(0).bloom_filter_length == 2064
+
+
+def test_column_metadata_with_index_page(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'nan_in_stats.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is True
+ assert metadata.row_group(0).column(0).index_page_offset == 0
Review Comment:
Feels weird that `has_index_page` is True and `index_page_offset` is 0 but this is a C++ issue (or a file issue) we are just acting as a pass-through here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #48201:
URL: https://github.com/apache/arrow/pull/48201#discussion_r2560814794
##########
python/pyarrow/tests/parquet/test_metadata.py:
##########
@@ -814,3 +816,31 @@ def msg(c):
with pytest.raises(TypeError, match=msg("FileMetaData")):
pq.FileMetaData()
+
+
+def test_column_metadata_with_bloom_filter(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'data_index_bloom_encoding_with_length.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is False
+ assert metadata.row_group(0).column(0).index_page_offset is None
+ assert metadata.row_group(0).column(0).has_bloom_filter is True
+ assert metadata.row_group(0).column(0).has_column_index is True
+ assert metadata.row_group(0).column(0).has_offset_index is True
+ assert metadata.row_group(0).column(0).bloom_filter_offset == 253
+ assert metadata.row_group(0).column(0).bloom_filter_length == 2064
+
+
+def test_column_metadata_with_index_page(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'nan_in_stats.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is True
+ assert metadata.row_group(0).column(0).index_page_offset == 0
Review Comment:
ok, I'll remove that :smiley:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48201:
URL: https://github.com/apache/arrow/pull/48201#issuecomment-3562832869
Revision: 26b9b638628b16b5d9525a12ef069013f4051d39
Submitted crossbow builds: [ursacomputing/crossbow @ actions-e4a9ff08ad](https://github.com/ursacomputing/crossbow/branches/all?query=actions-e4a9ff08ad)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/19570546117/job/56042767518)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545568/job/56042765447)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545201/job/56042764454)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545884/job/56042766706)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/19570546338/job/56042768050)|
|test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545367/job/56042764896)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545930/job/56042766955)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545714/job/56042765981)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/19570544941/job/56042763836)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545359/job/56042764828)|
|test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545628/job/56042765749)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545956/job/56042767057)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545803/job/56042766305)|
|test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/19570546432/job/56042768315)|
|test-conda-python-3.12-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545528/job/56042777838)|
|test-conda-python-3.12-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545105/job/56042764469)|
|test-conda-python-3.13|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545941/job/56042766797)|
|test-conda-python-3.13-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545087/job/56042764287)|
|test-conda-python-3.13-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/19570546268/job/56042768356)|
|test-conda-python-3.14|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545414/job/56042765006)|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545429/job/56042765074)|
|test-cuda-python-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545646/job/56042765669)|
|test-debian-12-python-3-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/19570544934/job/56042763867)|
|test-debian-12-python-3-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/19570546204/job/56042767713)|
|test-fedora-42-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/19570546210/job/56042767705)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/19570544913/job/56042763849)|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/19570546145/job/56042767456)|
|test-ubuntu-24.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/19570545019/job/56042763990)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #48201:
URL: https://github.com/apache/arrow/pull/48201#discussion_r2549592190
##########
python/pyarrow/tests/parquet/test_metadata.py:
##########
@@ -814,3 +816,31 @@ def msg(c):
with pytest.raises(TypeError, match=msg("FileMetaData")):
pq.FileMetaData()
+
+
+def test_column_metadata_with_bloom_filter(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'data_index_bloom_encoding_with_length.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is False
+ assert metadata.row_group(0).column(0).index_page_offset is None
+ assert metadata.row_group(0).column(0).has_bloom_filter is True
+ assert metadata.row_group(0).column(0).has_column_index is True
+ assert metadata.row_group(0).column(0).has_offset_index is True
+ assert metadata.row_group(0).column(0).bloom_filter_offset == 253
+ assert metadata.row_group(0).column(0).bloom_filter_length == 2064
+
+
+def test_column_metadata_with_index_page(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'nan_in_stats.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is True
+ assert metadata.row_group(0).column(0).index_page_offset == 0
Review Comment:
Feels weird that `has_index_page` is True and `index_page_offset` is 0 but this is a C++ issue we are just acting as a pass-through here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48201:
URL: https://github.com/apache/arrow/pull/48201#discussion_r2560603114
##########
python/pyarrow/tests/parquet/test_metadata.py:
##########
@@ -814,3 +816,31 @@ def msg(c):
with pytest.raises(TypeError, match=msg("FileMetaData")):
pq.FileMetaData()
+
+
+def test_column_metadata_with_bloom_filter(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'data_index_bloom_encoding_with_length.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is False
+ assert metadata.row_group(0).column(0).index_page_offset is None
+ assert metadata.row_group(0).column(0).has_bloom_filter is True
+ assert metadata.row_group(0).column(0).has_column_index is True
+ assert metadata.row_group(0).column(0).has_offset_index is True
+ assert metadata.row_group(0).column(0).bloom_filter_offset == 253
+ assert metadata.row_group(0).column(0).bloom_filter_length == 2064
+
+
+def test_column_metadata_with_index_page(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'nan_in_stats.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is True
+ assert metadata.row_group(0).column(0).index_page_offset == 0
Review Comment:
Index pages don't actually exist, so it seems a bit confusing to expose this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #48201:
URL: https://github.com/apache/arrow/pull/48201#issuecomment-3562823849
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #48201:
URL: https://github.com/apache/arrow/pull/48201#discussion_r2560625478
##########
python/pyarrow/tests/parquet/test_metadata.py:
##########
@@ -814,3 +816,31 @@ def msg(c):
with pytest.raises(TypeError, match=msg("FileMetaData")):
pq.FileMetaData()
+
+
+def test_column_metadata_with_bloom_filter(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'data_index_bloom_encoding_with_length.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is False
+ assert metadata.row_group(0).column(0).index_page_offset is None
+ assert metadata.row_group(0).column(0).has_bloom_filter is True
+ assert metadata.row_group(0).column(0).has_column_index is True
+ assert metadata.row_group(0).column(0).has_offset_index is True
+ assert metadata.row_group(0).column(0).bloom_filter_offset == 253
+ assert metadata.row_group(0).column(0).bloom_filter_length == 2064
+
+
+def test_column_metadata_with_index_page(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'nan_in_stats.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is True
+ assert metadata.row_group(0).column(0).index_page_offset == 0
Review Comment:
I thought they were deprecated but some old parquet files might have them? Isn't that the case? This was my understanding from reading the thrift spec. I can remove that from PyArrow if you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48201:
URL: https://github.com/apache/arrow/pull/48201#discussion_r2560708747
##########
python/pyarrow/tests/parquet/test_metadata.py:
##########
@@ -814,3 +816,31 @@ def msg(c):
with pytest.raises(TypeError, match=msg("FileMetaData")):
pq.FileMetaData()
+
+
+def test_column_metadata_with_bloom_filter(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'data_index_bloom_encoding_with_length.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is False
+ assert metadata.row_group(0).column(0).index_page_offset is None
+ assert metadata.row_group(0).column(0).has_bloom_filter is True
+ assert metadata.row_group(0).column(0).has_column_index is True
+ assert metadata.row_group(0).column(0).has_offset_index is True
+ assert metadata.row_group(0).column(0).bloom_filter_offset == 253
+ assert metadata.row_group(0).column(0).bloom_filter_length == 2064
+
+
+def test_column_metadata_with_index_page(parquet_test_datadir):
+ metadata = pq.read_metadata(parquet_test_datadir /
+ 'nan_in_stats.parquet')
+ assert metadata.row_group(0).column(0).has_dictionary_page is True
+ assert metadata.row_group(0).column(0).dictionary_page_offset == 4
+ assert metadata.row_group(0).column(0).has_index_page is True
+ assert metadata.row_group(0).column(0).index_page_offset == 0
Review Comment:
Well, perhaps they emitted this: https://github.com/apache/parquet-format/blob/3ab52ff2e4e1cbe4c52a3e25c0512803e860c454/src/main/thrift/parquet.thrift#L694-L696
:upside_down_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48199: [Python][Parquet] Expose existing Parquet C++ metadata about index_page and bloom_filter to Python [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48201:
URL: https://github.com/apache/arrow/pull/48201#issuecomment-3562791645
:warning: GitHub issue #48199 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org