You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2024/12/12 13:42:12 UTC
[PR] Add more integration tests [iceberg-rust]
Fokko opened a new pull request, #786:
URL: https://github.com/apache/iceberg-rust/pull/786
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Allow for schema evolution [iceberg-rust]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #786:
URL: https://github.com/apache/iceberg-rust/pull/786#issuecomment-2540869512
This is pretty similar. With this PR we first write an int, and then read it as a long, which is a valid schema evolution. In the case of #790 we try to read a narrower decimal from the Parquet file, into a wider one from the Iceberg schema. I haven't tested it with the decimal, that might be a good followup 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Allow for schema evolution [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #786:
URL: https://github.com/apache/iceberg-rust/pull/786#issuecomment-2540855953
Hi @Fokko, I'm following up on issue #790.
It appears this PR doesn't address the scenario I described in #790, where the underlying parquet schema differs from the declared one. Could schema evolution potentially resolve this? I'm concerned we might also need to adjust the Arrow schema mappings instead of just project mask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Allow for schema evolution [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #786:
URL: https://github.com/apache/iceberg-rust/pull/786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add more integration tests [iceberg-rust]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #786:
URL: https://github.com/apache/iceberg-rust/pull/786#discussion_r1882162789
##########
crates/integration_tests/tests/read_evolved_schema.rs:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Integration tests for rest catalog.
+
+use arrow_array::{Int64Array, StringArray};
+use futures::{StreamExt, TryStreamExt};
+use iceberg::expr::Reference;
+use iceberg::spec::Datum;
+use iceberg::{Catalog, TableIdent};
+use iceberg_integration_tests::set_test_fixture;
+use std::fmt::Pointer;
+
+
+#[tokio::test]
+async fn test_evolved_schema() {
+ let fixture = set_test_fixture("read_evolved_schema").await;
+
+ let catalog = fixture.rest_catalog;
+
+ let table = catalog
+ .load_table(
+ &TableIdent::from_strs(["default", "test_rename_column"])
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+
+ let predicate =
+ Reference::new("language").not_equal_to(Datum::string("Rust"));
+
+ let scan = table.scan().with_filter(predicate).build();
+ let batch_stream = scan.unwrap().to_arrow().await.unwrap();
+
+ let batches: Vec<_> = batch_stream.try_collect().await.unwrap();
+
+ let batch0 = batches[0].column_by_name("language").unwrap();
+ let batch1 = batches[1].column_by_name("language").unwrap();
+
+ assert_eq!(batch0.as_any().downcast_ref::<StringArray>().unwrap().value(0), "Java");
+ assert_eq!(batch1.as_any().downcast_ref::<StringArray>().unwrap().value(0), "Python");
+
+ // Evolve column - Does not yet work
Review Comment:
With the rule added above, it still fails with:
```
Parquet schema table {
1: foo: required int
}
and Iceberg schema table {
1: foo: optional long
}
do not match.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Allow for schema evolution [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #786:
URL: https://github.com/apache/iceberg-rust/pull/786#issuecomment-2540873200
Thank you for the explanation! This PR is good enough for me. Let's proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org