You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ajantha-bhat (via GitHub)" <gi...@apache.org> on 2024/01/18 12:30:54 UTC
[I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]
ajantha-bhat opened a new issue, #9511:
URL: https://github.com/apache/iceberg/issues/9511
build: https://github.com/apache/iceberg/actions/runs/7569167293/job/20611887361
PR: https://github.com/apache/iceberg/pull/9507
position deletes are missing? cache?
```
TestSparkExecutorCache > testMergeOnReadDelete() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, io-impl=org.apache.iceberg.spark.TestSparkExecutorCache$CustomFileIO, default-namespace=default} FAILED
java.lang.AssertionError:
Expecting all elements of:
[GenericDeleteFile{content=position_deletes, file_path=/tmp/junit17911827767769921495/pos-deletes-da2fc9e8-2fc7-42ad-901b-9139ee8102b0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=1267, column_sizes={2147483546=145, 2147483545=69}, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=null, upper_bounds=null, key_metadata=null, split_offsets=[4], equality_ids=null, sort_order_id=null, data_sequence_number=null, file_sequence_number=null},
GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17911827767769921495/eq-deletes-15c2d19c-c3dd-4b17-95cc-7e68759550f0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}]
to match given predicate but this element did not:
GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17911827767769921495/eq-deletes-15c2d19c-c3dd-4b17-95cc-7e68759550f0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}
at org.apache.iceberg.spark.TestSparkExecutorCache.checkDelete(TestSparkExecutorCache.java:268)
at org.apache.iceberg.spark.TestSparkExecutorCache.testMergeOnReadDelete(TestSparkExecutorCache.java:254)
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #9511:
URL: https://github.com/apache/iceberg/issues/9511#issuecomment-2302472715
It looks this is happening again https://github.com/apache/iceberg/actions/runs/10486032923/job/29043393618
```
TestSparkExecutorCache > testMergeOnReadUpdate() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, io-impl=org.apache.iceberg.spark.TestSparkExecutorCache$CustomFileIO, default-namespace=default} FAILED
java.lang.AssertionError:
Expecting all elements of:
[GenericDeleteFile{content=position_deletes, file_path=/tmp/junit17759035431822043198/pos-deletes-bb0e7a65-0665-4aca-8420-4b1d350158fb, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=1267, column_sizes={2147483546=147, 2147483545=69}, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=null, upper_bounds=null, key_metadata=null, split_offsets=[4], equality_ids=null, sort_order_id=null, data_sequence_number=null, file_sequence_number=null},
GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17759035431822043198/eq-deletes-44c2c956-e9c2-4a5f-8d99-8cea6fdfa6a4, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}]
to match given predicate but this element did not:
GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17759035431822043198/eq-deletes-44c2c956-e9c2-4a5f-8d99-8cea6fdfa6a4, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}
at org.apache.iceberg.spark.TestSparkExecutorCache.checkUpdate(TestSparkExecutorCache.java:303)
at org.apache.iceberg.spark.TestSparkExecutorCache.testMergeOnReadUpdate(TestSparkExecutorCache.java:286)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #9511:
URL: https://github.com/apache/iceberg/issues/9511#issuecomment-1900672359
The error says the equality delete file doesn't match the predicate, ie `streamCount(deleteFile) <= 1`.
```java
int maxRequestCount = mode == COPY_ON_WRITE ? 3 : 1;
assertThat(deleteFiles).allMatch(deleteFile -> streamCount(deleteFile) <= maxRequestCount);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on issue #9511:
URL: https://github.com/apache/iceberg/issues/9511#issuecomment-1912791701
I'll look into it, probably related to some cache invalidation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org