You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "drexler-sky (via GitHub)" <gi...@apache.org> on 2026/05/12 17:14:19 UTC
[PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
drexler-sky opened a new pull request, #16303:
URL: https://github.com/apache/iceberg/pull/16303
backport https://github.com/apache/iceberg/pull/16295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#issuecomment-4447073043
Thanks for fixing the imports @drexler-sky
the CI failing was actually a legitimate issue. Seems like the test ran in a loop until GitHub reports No space left on device
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao closed pull request #16303: Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering
URL: https://github.com/apache/iceberg/pull/16303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao merged PR #16303:
URL: https://github.com/apache/iceberg/pull/16303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra closed pull request #16303: Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering
URL: https://github.com/apache/iceberg/pull/16303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#issuecomment-4448222748
Thanks, everyone!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#issuecomment-4446043280
not sure why this particular pr fails CI, merge latest main into it. lets see if that works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "aihuaxu (via GitHub)" <gi...@apache.org>.
aihuaxu commented on code in PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#discussion_r3238571814
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java:
##########
@@ -118,26 +117,11 @@ public NamedReference[] filterAttributes() {
}
@Override
- public void filter(Filter[] filters) {
- Preconditions.checkState(
- Objects.equals(snapshotId(), currentSnapshotId()),
- "Runtime file filtering is not possible: the table has been concurrently modified. "
- + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
- + "If an external process modifies the table, enable table caching in the catalog. "
- + "If multiple threads modify the table, use independent Spark sessions in each thread.",
- snapshotId(),
- currentSnapshotId());
-
Review Comment:
It's removed in https://github.com/apache/iceberg/pull/15240/changes#diff-417c91abd0a86688c3f4f45fba4e80aac9e221bc8e7aa1a59e74ce27ce2222c5. We need to keep import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#discussion_r3238387587
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java:
##########
@@ -118,26 +117,11 @@ public NamedReference[] filterAttributes() {
}
@Override
- public void filter(Filter[] filters) {
- Preconditions.checkState(
- Objects.equals(snapshotId(), currentSnapshotId()),
- "Runtime file filtering is not possible: the table has been concurrently modified. "
- + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
- + "If an external process modifies the table, enable table caching in the catalog. "
- + "If multiple threads modify the table, use independent Spark sessions in each thread.",
- snapshotId(),
- currentSnapshotId());
-
Review Comment:
as @huaxingao points out offline, spark 4.1 does not currently have this guard
https://github.com/apache/iceberg/blame/e5e2e8b610af408ea33613d2739ccf39428cbe2a/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java#L95-L96
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#discussion_r3238398335
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java:
##########
@@ -118,26 +117,11 @@ public NamedReference[] filterAttributes() {
}
@Override
- public void filter(Filter[] filters) {
- Preconditions.checkState(
- Objects.equals(snapshotId(), currentSnapshotId()),
- "Runtime file filtering is not possible: the table has been concurrently modified. "
- + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
- + "If an external process modifies the table, enable table caching in the catalog. "
- + "If multiple threads modify the table, use independent Spark sessions in each thread.",
- snapshotId(),
- currentSnapshotId());
-
- for (Filter filter : filters) {
- // Spark can only pass In filters at the moment
- if (filter instanceof In
- && ((In) filter).attribute().equalsIgnoreCase(MetadataColumns.FILE_PATH.name())) {
- In in = (In) filter;
-
- Set<String> fileLocations = Sets.newHashSet();
- for (Object value : in.values()) {
- fileLocations.add((String) value);
- }
+ public void filter(Predicate[] predicates) {
+ for (Predicate predicate : predicates) {
Review Comment:
```suggestion
Preconditions.checkState(
Objects.equals(snapshotId(), currentSnapshotId()),
"Runtime file filtering is not possible: the table has been concurrently modified. "
+ "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
+ "If an external process modifies the table, enable table caching in the catalog. "
+ "If multiple threads modify the table, use independent Spark sessions in each thread.",
snapshotId(),
currentSnapshotId());
for (Predicate predicate : predicates) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#discussion_r3238402283
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java:
##########
@@ -118,26 +117,11 @@ public NamedReference[] filterAttributes() {
}
@Override
- public void filter(Filter[] filters) {
- Preconditions.checkState(
- Objects.equals(snapshotId(), currentSnapshotId()),
- "Runtime file filtering is not possible: the table has been concurrently modified. "
- + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
- + "If an external process modifies the table, enable table caching in the catalog. "
- + "If multiple threads modify the table, use independent Spark sessions in each thread.",
- snapshotId(),
- currentSnapshotId());
-
- for (Filter filter : filters) {
- // Spark can only pass In filters at the moment
- if (filter instanceof In
- && ((In) filter).attribute().equalsIgnoreCase(MetadataColumns.FILE_PATH.name())) {
- In in = (In) filter;
-
- Set<String> fileLocations = Sets.newHashSet();
- for (Object value : in.values()) {
- fileLocations.add((String) value);
- }
+ public void filter(Predicate[] predicates) {
+ for (Predicate predicate : predicates) {
Review Comment:
```suggestion
Preconditions.checkState(
Objects.equals(snapshotId(), currentSnapshotId()),
"Runtime file filtering is not possible: the table has been concurrently modified. "
+ "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
+ "If an external process modifies the table, enable table caching in the catalog. "
+ "If multiple threads modify the table, use independent Spark sessions in each thread.",
snapshotId(),
currentSnapshotId());
for (Predicate predicate : predicates) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#discussion_r3238364774
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java:
##########
@@ -118,26 +117,11 @@ public NamedReference[] filterAttributes() {
}
@Override
- public void filter(Filter[] filters) {
- Preconditions.checkState(
- Objects.equals(snapshotId(), currentSnapshotId()),
- "Runtime file filtering is not possible: the table has been concurrently modified. "
- + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
- + "If an external process modifies the table, enable table caching in the catalog. "
- + "If multiple threads modify the table, use independent Spark sessions in each thread.",
- snapshotId(),
- currentSnapshotId());
-
Review Comment:
looks like this is actually not a clean backport...
https://github.com/apache/iceberg/pull/16295/files#diff-417c91abd0a86688c3f4f45fba4e80aac9e221bc8e7aa1a59e74ce27ce2222c5L95-L105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#discussion_r3238403855
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkCopyOnWriteScan.java:
##########
@@ -103,26 +102,11 @@ public NamedReference[] filterAttributes() {
}
@Override
- public void filter(Filter[] filters) {
- Preconditions.checkState(
- Objects.equals(snapshotId(), currentSnapshotId()),
- "Runtime file filtering is not possible: the table has been concurrently modified. "
- + "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
- + "If an external process modifies the table, enable table caching in the catalog. "
- + "If multiple threads modify the table, use independent Spark sessions in each thread.",
- snapshotId(),
- currentSnapshotId());
-
- for (Filter filter : filters) {
- // Spark can only pass In filters at the moment
- if (filter instanceof In
- && ((In) filter).attribute().equalsIgnoreCase(MetadataColumns.FILE_PATH.name())) {
- In in = (In) filter;
-
- Set<String> fileLocations = Sets.newHashSet();
- for (Object value : in.values()) {
- fileLocations.add((String) value);
- }
+ public void filter(Predicate[] predicates) {
+ for (Predicate predicate : predicates) {
Review Comment:
```suggestion
Preconditions.checkState(
Objects.equals(snapshotId(), currentSnapshotId()),
"Runtime file filtering is not possible: the table has been concurrently modified. "
+ "Row-level operation scan snapshot ID: %s, current table snapshot ID: %s. "
+ "If an external process modifies the table, enable table caching in the catalog. "
+ "If multiple threads modify the table, use independent Spark sessions in each thread.",
snapshotId(),
currentSnapshotId());
for (Predicate predicate : predicates) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao closed pull request #16303: Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering
URL: https://github.com/apache/iceberg/pull/16303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport migrate SparkCopyOnWriteScan to SupportsRuntimeV2Filtering [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #16303:
URL: https://github.com/apache/iceberg/pull/16303#issuecomment-4443656478
ugh i had to retrigger this 3 times already, let me see why CI is not happy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org