You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "nathanb9 (via GitHub)" <gi...@apache.org> on 2026/04/07 23:53:01 UTC
[PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
nathanb9 opened a new pull request, #9673:
URL: https://github.com/apache/arrow-rs/pull/9673
## Rationale for this change
`ParquetMetaDataPushDecoder` clears exact requested ranges, but larger
speculative pushed ranges can remain buffered in `PushBuffers`. This
adds a way for callers to explicitly release non exact ranges
## What changes are included in this PR?
This adds `clear_all_ranges()`, which clears all byte ranges still
staged in the decoder's internal `PushBuffers`
## Are these changes tested?
Kinda tested. Tests added to verify the buffer is empty after clearing.
## Are there any user-facing changes?
Yes, this adds a new public `clear_all_ranges()` API on
`ParquetMetaDataPushDecoder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3048630874
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
Not useful i think.
Also, how does the CI work? Does it use the pr label to figure out which feature flags to use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3048630874
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
Not useful i think.
Also, how does the CI work? Does it use the pr table to figure out the fearture flags?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3075627810
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
will restart it -- thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3069740920
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
I added that flag in previous [pr](https://github.com/apache/arrow-rs/pull/9624/changes). The reason I want to remove it now is bc the flag causes failures in dev pr because we do not make change to a file in the arrow dir. The dev pr tests figures out which flag based on directory edited: https://github.com/apache/arrow-rs/blob/main/.github/workflows/dev_pr/labeler.yml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3073947156
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
got it. Makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3072969394
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
You can see ci in https://github.com/apache/arrow-rs/blob/main/.github/workflows/parquet.yml
It checks a variety of feature combinations: https://github.com/apache/arrow-rs/blob/68851ef953fd771cc310203c446e54145d4407e1/.github/workflows/parquet.yml#L96-L125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3069740920
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
I added that flag in previous [pr](https://github.com/apache/arrow-rs/pull/9624/changes). The reason I want to remove it now is bc the flag causes failures in dev pr because we do not make change to a file in the arrow dir. The dev pr figures out which flag based on directory edited: https://github.com/apache/arrow-rs/blob/main/.github/workflows/dev_pr/labeler.yml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3059998068
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
Well, the feature gate is necessary without the changes in this PR 😉.
IIRC the CI will do builds with various sets of features enabled. The actual unit tests are then run with default features and all features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3075100094
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
@alamb FYI incase it matters the CI failed in main. It looks like a transient network thing not related to above but not sure. Retry might fix?
logs show:
```...1: error sending request for url (https://static.rust-lang.org/dist/channel-rust-1.85.0.toml.sha256)\n 2: client error (Connect)\n 3: tcp connect error: Network is unreachable (os error 101)\n 4: Network is unreachable (os error 101)\n...```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#discussion_r3048630874
##########
parquet/src/util/push_buffers.rs:
##########
@@ -156,7 +156,6 @@ impl PushBuffers {
}
/// Clear all buffered ranges and their corresponding data
- #[cfg(feature = "arrow")]
Review Comment:
Not useful i think.
Also, how does the CI work? Does it use the pr label to figure out the which feature flags to use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673#issuecomment-4231932515
@etseidl tysm. I added the motivating ticket which was create by alamb@ in the PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ParquetMetaDataPushDecoder API to clear all buffered ranges [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9673:
URL: https://github.com/apache/arrow-rs/pull/9673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org