You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2024/01/14 13:36:23 UTC
[PR] ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17 [orc]
cxzl25 opened a new pull request, #1748:
URL: https://github.com/apache/orc/pull/1748
### What changes were proposed in this pull request?
Add java options `--add-opens=java.base/sun.nio.ch=ALL-UNNAMED`
### Why are the changes needed?
```java
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x5b2c883c) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x5b2c883c
at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:121)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:358)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:295)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:344)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:196)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:284)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:483)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
at org.apache.orc.bench.spark.SparkBenchmark$InputSource.setup(SparkBenchmark.java:129)
at org.apache.orc.bench.spark.jmh_generated.SparkBenchmark_fullRead_jmhTest._jmh_tryInit_f_inputsource1_1(SparkBenchmark_fullRead_jmhTest.java:403)
```
### How was this patch tested?
local test
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17 [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1748: ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17
URL: https://github.com/apache/orc/pull/1748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17 [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1748:
URL: https://github.com/apache/orc/pull/1748#issuecomment-1894021983
Ya, sorry, I realized that @cxzl25 already mentioned the above pom.xml. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17 [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1748:
URL: https://github.com/apache/orc/pull/1748#issuecomment-1891302043
> Not very clear at what point are we hitting this error
Because Spark uses Unsafe API.
https://github.com/apache/spark/blob/branch-3.5/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala#L207-L213
https://github.com/apache/spark/blob/416b7b1cd5a6555a2d545d2f8f3cbd6cadff130e/pom.xml#L304-L322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17 [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1748:
URL: https://github.com/apache/orc/pull/1748#issuecomment-1894048023
Merged to main/2.0/1.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1586: Fix IllegalAccessError when SparkBenchmark runs on JDK17 [orc]
Posted by "mystic-lama (via GitHub)" <gi...@apache.org>.
mystic-lama commented on PR #1748:
URL: https://github.com/apache/orc/pull/1748#issuecomment-1891298062
Not very clear at what point are we hitting this error? Could you please elaborate the description a bit more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org