You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huaxingao (via GitHub)" <gi...@apache.org> on 2026/01/03 03:02:27 UTC
[PR] Spark: Fix flaky ordering in Spark TestSelect [iceberg]
huaxingao opened a new pull request, #14956:
URL: https://github.com/apache/iceberg/pull/14956
Fix flaky ordering in Spark `TestSelect.testSelectWithSpecifiedTargetSplitSize()`
```
TestRemoteScanPlanning > testSelectWithSpecifiedTargetSplitSize() > catalogName = testrest, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=rest, cache-enabled=false, uri=http://localhost:36105/, rest-scan-planning-enabled=true}, binaryTableName = testrest.default.binary_table FAILED
org.opentest4j.AssertionFailedError: [Should return all expected rows: row 1 contents should match]
expected: 1L
but was: 2L
at app//org.apache.iceberg.spark.SparkTestHelperBase.assertEquals(SparkTestHelperBase.java:91)
at app//org.apache.iceberg.spark.SparkTestHelperBase.assertEquals(SparkTestHelperBase.java:67)
at app//org.apache.iceberg.spark.sql.TestSelect.testSelectWithSpecifiedTargetSplitSize(TestSelect.java:124)
```
Also fix a few other places that could cause ordering problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Fix flaky ordering in Spark TestSelect [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 merged PR #14956:
URL: https://github.com/apache/iceberg/pull/14956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Fix flaky ordering in Spark TestSelect [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #14956:
URL: https://github.com/apache/iceberg/pull/14956#discussion_r2658677013
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestSelect.java:
##########
@@ -155,22 +157,19 @@ public void testSelectRewrite() {
@TestTemplate
public void selectWithLimit() {
- Object[] first = row(1L, "a", 1.0F);
- Object[] second = row(2L, "b", 2.0F);
- Object[] third = row(3L, "c", Float.NaN);
-
- // verify that LIMIT is properly applied in case SupportsPushDownLimit.isPartiallyPushed() is
- // ever overridden in SparkScanBuilder
- assertThat(sql("SELECT * FROM %s LIMIT 1", tableName)).containsExactly(first);
- assertThat(sql("SELECT * FROM %s LIMIT 2", tableName)).containsExactly(first, second);
- assertThat(sql("SELECT * FROM %s LIMIT 3", tableName)).containsExactly(first, second, third);
+ // Note: without ORDER BY, the specific rows returned are not deterministic, especially when
+ // remote scan planning or split planning changes the physical scan order. This test only
+ // asserts that the LIMIT is enforced.
Review Comment:
i would still recommend checking the records to make sure same record is not returned twice ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Fix flaky ordering in Spark TestSelect [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #14956:
URL: https://github.com/apache/iceberg/pull/14956#discussion_r2658728509
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestSelect.java:
##########
@@ -155,22 +157,19 @@ public void testSelectRewrite() {
@TestTemplate
public void selectWithLimit() {
- Object[] first = row(1L, "a", 1.0F);
- Object[] second = row(2L, "b", 2.0F);
- Object[] third = row(3L, "c", Float.NaN);
-
- // verify that LIMIT is properly applied in case SupportsPushDownLimit.isPartiallyPushed() is
- // ever overridden in SparkScanBuilder
- assertThat(sql("SELECT * FROM %s LIMIT 1", tableName)).containsExactly(first);
- assertThat(sql("SELECT * FROM %s LIMIT 2", tableName)).containsExactly(first, second);
- assertThat(sql("SELECT * FROM %s LIMIT 3", tableName)).containsExactly(first, second, third);
+ // Note: without ORDER BY, the specific rows returned are not deterministic, especially when
+ // remote scan planning or split planning changes the physical scan order. This test only
+ // asserts that the LIMIT is enforced.
Review Comment:
Make sense. I change to `order by`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org