You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2024/12/09 07:04:35 UTC
[PR] Bump Spark 3.5.4 RC1 [iceberg]
pan3793 opened a new pull request, #11731:
URL: https://github.com/apache/iceberg/pull/11731
https://lists.apache.org/thread/btdsos7b92962lh10hzswkfy3oj1v0fb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2693720164
@Fokko @nastra, May I request that release 1.6.2 ship this patch to allow the users who are stuck with Java 8 to use the latest Spark 3.5.x with Iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1893993352
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,12 @@ public void close() {
accessor.close();
}
+ @Override
Review Comment:
> Did we land on adding the `@Override`? Now it doesn't compile against Spark 3.5.2. Leaving out the annotation is safer (it will still work with ≥3.5.4).
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1879272199
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,11 @@ public void close() {
accessor.close();
}
+ // If a column vector is writable or constant, it should override this method and do nothing.
+ // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+ @Override
+ public void closeIfNotWritable() {}
Review Comment:
I checked the codebase, other vectors overwrite `void close() {}`, so they are not affected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #11731:
URL: https://github.com/apache/iceberg/pull/11731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2532346946
Please check for some discussion there https://github.com/apache/spark/pull/49131#issuecomment-2532341673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "gunrk (via GitHub)" <gi...@apache.org>.
gunrk commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2898639874
@pan3793 Can you confirm if this was ever patched to 1.6.x? I don't see a 1.6.2 release.
We face similar segv errors in our spark executors after upgrading to spark 3.5.4 with iceberg 1.6.1
```
# A fatal error has been detected by the Java Runtime Environment:
--
# SIGSEGV (0xb) at pc=0x00007f047c7f1511, pid=1, tid=0x00007ef4069e9700
```
We see IlegalAccessError and the process exits with code 139.
Unfortunately we are on Java 8 and It seems Iceberg 1.7 removed support for the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1879240394
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,11 @@ public void close() {
accessor.close();
}
+ // If a column vector is writable or constant, it should override this method and do nothing.
+ // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+ @Override
+ public void closeIfNotWritable() {}
Review Comment:
will rename to `closeIfFreeable` after RC2 available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1879272199
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,11 @@ public void close() {
accessor.close();
}
+ // If a column vector is writable or constant, it should override this method and do nothing.
+ // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+ @Override
+ public void closeIfNotWritable() {}
Review Comment:
I checked the codebase, other vectors overwrite `void close() {}`, so they won't be affected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2530114142
the upgrading causes a core dump in iceberg-spark-3.5's tests
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f8529a34563, pid=83775, tid=83793
#
# JRE version: OpenJDK Runtime Environment Zulu17.54+21-CRaC-CA (17.0.13+11) (build 17.0.13+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.54+21-CRaC-CA (17.0.13+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0xc34563] ProtectionDomainEntry::object_no_keepalive()+0x3
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/iceberg/iceberg/spark/v3.5/spark/core.83775)
#
# An error report file with more information is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/hs_err_pid83775.log
#
# Compiler replay data is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/replay_pid83775.log
#
# If you would like to submit a bug report, please visit:
# http://www.azul.com/support/
#
> Task :iceberg-spark:iceberg-spark-3.5_2.12:test FAILED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2556920961
Spark 3.5.4 RC3 passed the vote and the jars were available on Maven Central a few minutes ago, I removed the staging repo and it's ready to go.
cc @nastra @Fokko and @jbonofre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC2 [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2548667713
Thanks @pan3793 for catching this early. Could you create a separate PR for just the patch? We can include this in 1.7.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2532122346
update: the failure is related to [SPARK-50235](https://github.com/apache/spark/pull/48767), and the test passed on Spark 3.5.4 RC0 with reverting that patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2536419878
Thank you for checking Apache Iceberg side, @pan3793 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2900000583
@gunrk no 1.6.2 yet, I think you can request a release on the mailing list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1893985933
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,12 @@ public void close() {
accessor.close();
}
+ @Override
Review Comment:
Did we land on adding the `@Override`? Now it doesn't compile against Spark 3.5.2. Leaving out the annotation is safer (it will still work with ≥3.5.4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2533597688
FYI @LuciferYang @viirya @dongjoon-hyun @wForget
CI is green now, and I think the current Spark `branch-3.5` is in good shape for Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2533468089
@viirya @LuciferYang thank you for the guidance, let me try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC2 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1889519453
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,11 @@ public void close() {
accessor.close();
}
+ // If a column vector is writable or constant, it should override this method and do nothing.
+ // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC2 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2548892617
@Fokko I opened #11802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1894023274
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,12 @@ public void close() {
accessor.close();
}
+ @Override
Review Comment:
removed.
both keeping and removing are fine to me, it has no difference in runtime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2532312814
Does Iceberg implement ColumnVector interface which is similar to writable column vector or constant column vector?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2532512457
> Please check for some discussion there [apache/spark#49131 (comment)](https://github.com/apache/spark/pull/49131#issuecomment-2532341673)
Yes, I tested it locally and found that after adding a no-op `closeIfFreeable` method to the `IcebergArrowColumnVector.java` in version 3.5, all the test cases in `TestRewriteDataFilesAction` were able to pass. We can further test it after the release of 3.5.4 RC2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1894021257
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,12 @@ public void close() {
accessor.close();
}
+ @Override
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC2 [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1888888902
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,11 @@ public void close() {
accessor.close();
}
+ // If a column vector is writable or constant, it should override this method and do nothing.
+ // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
Review Comment:
nit: can you please move the comment into the method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2569750175
Thanks @pan3793 for fixing this! And thanks @singhpk234, @huaxingao, @LuciferYang, and @viirya for reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#issuecomment-2530152795
the failure seems related to arrow
minimal reproduce command
```
./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 -DhiveVersions= -DflinkVersions= -DkafkaVersions= \
:iceberg-spark:iceberg-spark-3.5_2.12:test --tests=TestRewriteDataFilesAction \
-Pquick=true -x javadoc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Bump Spark 3.5.4 RC1 [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #11731:
URL: https://github.com/apache/iceberg/pull/11731#discussion_r1879265358
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java:
##########
@@ -59,6 +59,11 @@ public void close() {
accessor.close();
}
+ // If a column vector is writable or constant, it should override this method and do nothing.
+ // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+ @Override
+ public void closeIfNotWritable() {}
Review Comment:
> If a column vector is writable or constant, it should override this method and do nothing.
we have ConstantColumnVector as well, should we do it vector classes which absolutely requires it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org