You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gabotechs (via GitHub)" <gi...@apache.org> on 2026/09/21 15:14:37 UTC
[PR] Bump to Arrow 60 [iceberg-rust]
gabotechs opened a new pull request, #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## What changes are included in this PR?
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
Bumps the `arrow` dependency to 60, addressing all breaking API changes.
## Are these changes tested?
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Yes, by existing tests
## AI Disclosure
<!--
https://iceberg.apache.org/contribute/#guidelines-for-ai-assisted-contributions
-->
The code is fully generated by Codex and reviewed by me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "gabotechs (via GitHub) via issues" <is...@iceberg.apache.org>.
gabotechs commented on code in PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#discussion_r4522434638
##########
crates/iceberg/src/arrow/reader/row_filter.rs:
##########
Review Comment:
Good catch, thanks. The evaluator now borrows `RowGroupPageIndex` and looks up columns directly, without cloning their indexes. The row-count buffer also uses the page count now.
##########
crates/iceberg/src/expr/visitors/page_index_evaluator.rs:
##########
@@ -162,7 +160,11 @@ impl<'a> PageIndexEvaluator<'a> {
));
};
- let Some(column_index) = self.column_index.get(parquet_column_index) else {
+ let Some(column_index) = self
+ .column_index
+ .get(parquet_column_index)
+ .and_then(Option::as_ref)
+ else {
// This should not happen, but we fail soft anyway so that the scan is still
// successful, just a bit slower
Review Comment:
Agreed, this is an expected case. Updated the comment as suggested.
##########
crates/iceberg/src/expr/visitors/page_index_evaluator.rs:
##########
@@ -185,12 +187,15 @@ impl<'a> PageIndexEvaluator<'a> {
match self.row_count_cache.get(&parquet_column_index) {
Some(count) => count.clone(),
None => {
- let Some(offset_index) = self.offset_index.get(parquet_column_index) else {
- // if we have a column index, we should always have an offset index.
- return Err(Error::new(
- ErrorKind::Unexpected,
- format!("Missing offset index for field id {field_id}"),
- ));
+ let Some(offset_index) = self
+ .offset_index
+ .get(parquet_column_index)
+ .and_then(Option::as_ref)
+ else {
+ // A page index provider can load column and offset indexes
+ // independently. Without offsets, we cannot construct a row
+ // selection, so preserve correctness by skipping page pruning.
+ return self.select_all_rows();
Review Comment:
Added your suggested test. I also checked that changing the fallback to `skip_all_rows()` makes it fail.
##########
crates/iceberg/src/arrow/value.rs:
##########
@@ -1406,7 +1408,17 @@ mod test {
let map_key_builder = Int32Builder::new();
let map_value_builder = Int32Builder::new();
- let map_builder = MapBuilder::new(None, map_key_builder, map_value_builder);
+ // Arrow 60 defaults to `key` and `value`; this fixture's schema uses
+ // the Iceberg field names `keys` and `values`.
Review Comment:
You're right, those names come from the fixture. Updated the comment as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-5805304010
Nice PR, thanks @gabotechs looks like a smooth transition, are you planning to merge the update closer to the release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "dannycjones (via GitHub) via issues" <is...@iceberg.apache.org>.
dannycjones commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6771964265
I think we should keep the `reqwest` bump separate unless it simplifies the changes in a meaningful way (or some other benefit).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "AdamGS (via GitHub) via issues" <is...@iceberg.apache.org>.
AdamGS commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6767575585
If its easier, I'm happy to do that myself in a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6775901848
makes sense, i merged the reqwest 0.13 upgrade pr (#3666) and rebased this pr on latest main to include that change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "AdamGS (via GitHub) via issues" <is...@iceberg.apache.org>.
AdamGS commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6767071216
Could you also bump `reqwest` to 0.13? That will make upgrades to users that also use `datafusion` (starting from the upcoming 56) or `object_store` much easier 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "gabotechs (via GitHub) via issues" <is...@iceberg.apache.org>.
gabotechs commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6762410599
@kevinjqliu I see 0.11 was released with:
- Arrow 58 (https://crates.io/crates/iceberg/0.11.0/code/Cargo.toml#L60)
- DataFusion 54 (https://crates.io/crates/iceberg-datafusion/0.11.0/code/Cargo.toml#L55)
It'd be nice to have 0.12 contain:
- Arrow 60
- No datafusion dep
Unfortunately, having this project depend on Arrow 58 will render it unusable to most people already depending on the Arrow ecosystem, as major versions will conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6692064357
curious what is our strategy for iceberg crate version to datafusion version mapping?
i see 0.11.0 is using df 53 (https://github.com/apache/iceberg-rust/commit/477a1e525b4915895388a4f45557b825ea541ef2)
the next df upgrade to df 55 is on main but not released (https://github.com/apache/iceberg-rust/commit/665c64e48e8d33797ecb1a421f327edd9b024879#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542)
and this one is upgrading to arrow 60.
what should 0.12.0 release contain?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "AdamGS (via GitHub) via issues" <is...@iceberg.apache.org>.
AdamGS commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6768716937
FWIW - that will actually save a dependency here, as some underlying crates (like `opendal`) already pull `reqwest` 0.13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "mbutrovich (via GitHub) via issues" <is...@iceberg.apache.org>.
mbutrovich commented on code in PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#discussion_r4482965279
##########
crates/iceberg/src/arrow/reader/row_filter.rs:
##########
Review Comment:
Could `PageIndexEvaluator` borrow the row group's page index instead of taking cloned vectors? As written, this deep-clones every column's `ColumnIndexMetaData` and `OffsetIndexMetaData` for every selected row group, including columns the predicate never reads. Each clone copies the per-page min and max values, null pages, null counts, level histograms, and page locations. On `main` the evaluator borrowed these straight from the metadata ([row_filter.rs](https://github.com/apache/iceberg-rust/blob/f533cd6175bb42b5788537f589cd9450a3054d27/crates/iceberg/src/arrow/reader/row_filter.rs#L139-L146)), so this is new work per file on the scan path.
Parquet 60's [`RowGroupPageIndex`](https://github.com/apache/arrow-rs/blob/b391e5af7dff9f6909af45c01f7dddb68ba08c0f/parquet/src/file/metadata/page_index.rs#L268-L322) already has per-column `column_index(column_idx)` and `offset_index(column_idx)` lookups. If `PageIndexEvaluator` holds `page_index: &'a RowGroupPageIndex` in place of the two slices, the lookups in `calc_row_selection` become `self.page_index.column_index(parquet_column_index)` and `self.page_index.offset_index(parquet_column_index)`, and this call site becomes the suggestion below. The import is `parquet::file::metadata::page_index::RowGroupPageIndex`, because `parquet::file::metadata` doesn't re-export it. `calc_row_counts` sizes its `Vec` with `self.offset_index.len()`, which is the column count, so that would become `page_locations.len()`. In the tests, `get_test_metadata` and the copy in `eval_matches_no_rows_for_empty_row_group` reduce to `metadata.page_index_for_row_group(0)`.
```suggestion
let selections_for_page = PageIndexEvaluator::eval(
predicate,
&parquet_metadata.page_index_for_row_group(idx),
row_group_metadata,
field_id_map,
snapshot_schema,
)?;
```
##########
crates/iceberg/src/expr/visitors/page_index_evaluator.rs:
##########
@@ -185,12 +187,15 @@ impl<'a> PageIndexEvaluator<'a> {
match self.row_count_cache.get(&parquet_column_index) {
Some(count) => count.clone(),
None => {
- let Some(offset_index) = self.offset_index.get(parquet_column_index) else {
- // if we have a column index, we should always have an offset index.
- return Err(Error::new(
- ErrorKind::Unexpected,
- format!("Missing offset index for field id {field_id}"),
- ));
+ let Some(offset_index) = self
+ .offset_index
+ .get(parquet_column_index)
+ .and_then(Option::as_ref)
+ else {
+ // A page index provider can load column and offset indexes
+ // independently. Without offsets, we cannot construct a row
+ // selection, so preserve correctness by skipping page pruning.
+ return self.select_all_rows();
Review Comment:
On `main` a missing offset index returned `ErrorKind::Unexpected`, and here it selects every row. Could we add a test for this branch? No existing test reaches it: changing this `select_all_rows()` to `skip_all_rows()` leaves the existing `page_index_evaluator` tests passing. With the `RowGroupPageIndex` change above, a test along these lines covers it by dropping the offset index for `col_float` only:
```rust
#[test]
fn eval_selects_all_rows_when_column_lacks_offset_index() -> Result<()> {
let (metadata, _temp_file) = create_test_parquet_file()?;
let (iceberg_schema_ref, field_id_map) = build_iceberg_schema_and_field_map()?;
// Keep both column indexes, but the offset index only for col_string.
let page_index = metadata.page_index().unwrap();
let mut builder = page_index
.as_any()
.downcast_ref::<PageIndex>()
.unwrap()
.clone()
.into_builder();
builder.allocate_offset_indexes(1, 2);
builder.put_offset_index(page_index.offset_index(0, 1).unwrap().clone(), 0, 1);
let metadata = metadata
.as_ref()
.clone()
.into_builder()
.set_page_index(Some(Arc::new(builder.build())))
.build();
let filter = Reference::new("col_float")
.is_not_null()
.bind(iceberg_schema_ref.clone(), false)?;
let result = PageIndexEvaluator::eval(
&filter,
&metadata.page_index_for_row_group(0),
metadata.row_group(0),
&field_id_map,
iceberg_schema_ref.as_ref(),
)?;
assert_eq!(result, vec![RowSelector::select(4096)]);
Ok(())
}
```
With the full page index, the same predicate on this fixture returns `[skip(2048), select(2048)]` (`eval_is_not_null_dont_select_pages_with_all_nulls`), so the test tells the two paths apart.
##########
crates/iceberg/src/expr/visitors/page_index_evaluator.rs:
##########
@@ -162,7 +160,11 @@ impl<'a> PageIndexEvaluator<'a> {
));
};
- let Some(column_index) = self.column_index.get(parquet_column_index) else {
+ let Some(column_index) = self
+ .column_index
+ .get(parquet_column_index)
+ .and_then(Option::as_ref)
+ else {
// This should not happen, but we fail soft anyway so that the scan is still
// successful, just a bit slower
Review Comment:
Should this comment change now that `ColumnIndexMetaData::NONE` is gone? In parquet 60 a column chunk without a column index is left as `None` in the page index ([parser.rs](https://github.com/apache/arrow-rs/blob/b391e5af7dff9f6909af45c01f7dddb68ba08c0f/parquet/src/file/metadata/parser.rs#L311-L316)), so it lands in this branch. On `main` that case was the `NONE` arm in `apply_predicate_to_column_index` ([page_index_evaluator.rs](https://github.com/apache/iceberg-rust/blob/f533cd6175bb42b5788537f589cd9450a3054d27/crates/iceberg/src/expr/visitors/page_index_evaluator.rs#L265-L268)). A writer that skips page statistics for a column produces it, so this is an expected path now.
```suggestion
// The writer can omit the column index for a column chunk, for example
// when page statistics are disabled. Without it we can't prune pages.
```
##########
crates/iceberg/src/arrow/value.rs:
##########
@@ -1406,7 +1408,17 @@ mod test {
let map_key_builder = Int32Builder::new();
let map_value_builder = Int32Builder::new();
- let map_builder = MapBuilder::new(None, map_key_builder, map_value_builder);
+ // Arrow 60 defaults to `key` and `value`; this fixture's schema uses
+ // the Iceberg field names `keys` and `values`.
Review Comment:
Could this comment say the names come from the fixture rather than from Iceberg? iceberg-rust names map fields `key` and `value` ([datatypes.rs](https://github.com/apache/iceberg-rust/blob/f533cd6175bb42b5788537f589cd9450a3054d27/crates/iceberg/src/spec/datatypes.rs#L41-L43)), so "the Iceberg field names `keys` and `values`" points a reader the wrong way. `keys` and `values` are what the fixture's hand-written Arrow type spells out at lines 1444-1445, which is why the builder has to match them.
```suggestion
// Arrow 60 `MapBuilder` defaults to `key` and `value`, but this fixture's
// Arrow type names the map fields `keys` and `values`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "techgopal (via GitHub) via issues" <is...@iceberg.apache.org>.
techgopal commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6693524934
We'd like to see this land as well. We use the `iceberg` crate from a query engine that's on Arrow 60 / Parquet 60 (not DataFusion), and the version gap means we can't share Arrow data with iceberg-rust in-process without an FFI round trip. A release on Arrow 60 would let us drop that bridge. Happy to help test a release candidate against our TPC-H-over-Iceberg checks if that's useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6101555041
> This would be for apache/datafusion-iceberg only, right? I think we could move forward in this repository, with a follow-up Arrow+DF in that repo?
Oh yes, thats a good point, which generates some complexity with dependencies. We can move with Arrow 60, and datafusion-iceberg is already with DF 55
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6777267376
adding to merge queue 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "gabotechs (via GitHub) via issues" <is...@iceberg.apache.org>.
gabotechs commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6652521868
DataFusion is preparing for the 56 release, which depends on Arrow 60. As `iceberg-rust` is still on Arrow 59, it will make it incompatible with the DataFusion ecosystem because of conflicting dependencies, so it's starting to be more pressing to get this change in.
cc @dannycjones in case you have time for reviewing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6775964076
> We shouldn't really have to worry about DataFusion too much going forward. Ideally I think this crate should try to jump to new major arrow-rs versions pretty quickly. I think 0.12.0 should have Arrow-rs 60.x. 61.0.0 will be out next month, but I think we should shoot for an upgrade and release before that.
oh right, 🤦 i forgot we removed the datafusion dependency (i dont see it in https://github.com/apache/iceberg-rust/blob/b879b9626754501f025516ce0f0e9535fa6fd414/Cargo.toml anymore)
And this PR is for Arrow.
@mbutrovich @gabotechs think this is good to merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu merged PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6066231749
> Arrow/Parquet 60 has some pretty useful reader improvements around adaptive row-group pruning. I'm exploring live runtime pruning in the Iceberg reader, and the newer APIs make the implementation substantially cleaner and more effective.
we would prob need to consider DF 55 migration as well.
Let me try to create a PR to see how large the migration is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "unikdahal (via GitHub)" <gi...@apache.org>.
unikdahal commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-5981729158
@comphead @kevinjqliu would it make sense to merge this now?
Arrow/Parquet 60 has some pretty useful reader improvements around adaptive row-group pruning. I'm exploring live runtime pruning in the Iceberg reader, and the newer APIs make the implementation substantially cleaner and more effective.
If the bump is otherwise ready, it would be really helpful to have this on main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "AdamGS (via GitHub) via issues" <is...@iceberg.apache.org>.
AdamGS commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6772237817
No problem, I'll have a PR soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "mbutrovich (via GitHub) via issues" <is...@iceberg.apache.org>.
mbutrovich commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6772363950
> curious what is our strategy for iceberg crate version to datafusion version mapping?
>
> i see 0.11.0 is using df 53 ([477a1e5](https://github.com/apache/iceberg-rust/commit/477a1e525b4915895388a4f45557b825ea541ef2)) the next df upgrade to df 55 is on main but not released ([665c64e#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542](https://github.com/apache/iceberg-rust/commit/665c64e48e8d33797ecb1a421f327edd9b024879#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542))
>
> and this one is upgrading to arrow 60.
>
> what should 0.12.0 release contain?
We shouldn't really have to worry about DataFusion too much going forward. Ideally I think this crate should try to jump to new major arrow-rs versions pretty quickly. I think 0.12.0 should have Arrow-rs 60.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6779099557
thanks @gabotechs and everyone for the review ❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "gabotechs (via GitHub) via issues" <is...@iceberg.apache.org>.
gabotechs commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6767582270
Happy to do it in this PR yes, but would like confirmation from maintainers here in case they prefer to keep updates separate in different PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-5808844455
> are you planning to merge the update closer to the release
Whenever maintainers find more convenient, on my side no blockers to merge this as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump to Arrow 60 [iceberg-rust]
Posted by "dannycjones (via GitHub)" <gi...@apache.org>.
dannycjones commented on PR #3257:
URL: https://github.com/apache/iceberg-rust/pull/3257#issuecomment-6087318258
> > Arrow/Parquet 60 has some pretty useful reader improvements around adaptive row-group pruning. I'm exploring live runtime pruning in the Iceberg reader, and the newer APIs make the implementation substantially cleaner and more effective.
>
> we would prob need to consider DF 55 migration as well. Let me try to create a PR to see how large the migration is
This would be for apache/datafusion-iceberg only, right? I think we could move forward in this repository, with a follow-up Arrow+DF in that repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org