You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ajantha-bhat (via GitHub)" <gi...@apache.org> on 2024/01/18 12:30:54 UTC

[I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]

ajantha-bhat opened a new issue, #9511:
URL: https://github.com/apache/iceberg/issues/9511

   build: https://github.com/apache/iceberg/actions/runs/7569167293/job/20611887361
   PR: https://github.com/apache/iceberg/pull/9507
   
   position deletes are missing? cache? 
   
   ```
   TestSparkExecutorCache > testMergeOnReadDelete() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, io-impl=org.apache.iceberg.spark.TestSparkExecutorCache$CustomFileIO, default-namespace=default} FAILED
       java.lang.AssertionError: 
       Expecting all elements of:
         [GenericDeleteFile{content=position_deletes, file_path=/tmp/junit17911827767769921495/pos-deletes-da2fc9e8-2fc7-42ad-901b-9139ee8102b0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=1267, column_sizes={2147483546=145, 2147483545=69}, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=null, upper_bounds=null, key_metadata=null, split_offsets=[4], equality_ids=null, sort_order_id=null, data_sequence_number=null, file_sequence_number=null},
           GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17911827767769921495/eq-deletes-15c2d19c-c3dd-4b17-95cc-7e68759550f0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}]
       to match given predicate but this element did not:
         GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17911827767769921495/eq-deletes-15c2d19c-c3dd-4b17-95cc-7e68759550f0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}
           at org.apache.iceberg.spark.TestSparkExecutorCache.checkDelete(TestSparkExecutorCache.java:268)
           at org.apache.iceberg.spark.TestSparkExecutorCache.testMergeOnReadDelete(TestSparkExecutorCache.java:254)
   OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has 
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #9511:
URL: https://github.com/apache/iceberg/issues/9511#issuecomment-2302472715

   It looks this is happening again https://github.com/apache/iceberg/actions/runs/10486032923/job/29043393618
   ```
   
   TestSparkExecutorCache > testMergeOnReadUpdate() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, io-impl=org.apache.iceberg.spark.TestSparkExecutorCache$CustomFileIO, default-namespace=default} FAILED
       java.lang.AssertionError: 
       Expecting all elements of:
         [GenericDeleteFile{content=position_deletes, file_path=/tmp/junit17759035431822043198/pos-deletes-bb0e7a65-0665-4aca-8420-4b1d350158fb, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=1267, column_sizes={2147483546=147, 2147483545=69}, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=null, upper_bounds=null, key_metadata=null, split_offsets=[4], equality_ids=null, sort_order_id=null, data_sequence_number=null, file_sequence_number=null},
           GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17759035431822043198/eq-deletes-44c2c956-e9c2-4a5f-8d99-8cea6fdfa6a4, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}]
       to match given predicate but this element did not:
         GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17759035431822043198/eq-deletes-44c2c956-e9c2-4a5f-8d99-8cea6fdfa6a4, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}
           at org.apache.iceberg.spark.TestSparkExecutorCache.checkUpdate(TestSparkExecutorCache.java:303)
           at org.apache.iceberg.spark.TestSparkExecutorCache.testMergeOnReadUpdate(TestSparkExecutorCache.java:286)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]

Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #9511:
URL: https://github.com/apache/iceberg/issues/9511#issuecomment-1900672359

   The error says the equality delete file doesn't match the predicate, ie `streamCount(deleteFile) <= 1`.
   ```java
       int maxRequestCount = mode == COPY_ON_WRITE ? 3 : 1;
       assertThat(deleteFiles).allMatch(deleteFile -> streamCount(deleteFile) <= maxRequestCount);
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Flaky test: TestSparkExecutorCache > testMergeOnReadDelete() [iceberg]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on issue #9511:
URL: https://github.com/apache/iceberg/issues/9511#issuecomment-1912791701

   I'll look into it, probably related to some cache invalidation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org