You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2024/01/14 13:39:11 UTC
[PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
cxzl25 opened a new pull request, #1749:
URL: https://github.com/apache/orc/pull/1749
### What changes were proposed in this pull request?
Use shaded jar name
### Why are the changes needed?
```bash
java -jar spark/target/orc-benchmarks-spark-*.jar spark data
```
```
Error: Invalid or corrupt jarfile spark/target/orc-benchmarks-spark-2.1.0-SNAPSHOT-nohive.jar
```
### How was this patch tested?
local test
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1749:
URL: https://github.com/apache/orc/pull/1749#discussion_r1454536564
##########
java/CMakeLists.txt:
##########
@@ -97,7 +97,7 @@ add_test(
add_test(
NAME java-bench-spark-test
- COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar spark data -d sales -c zstd -f orc
+ COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}-shaded.jar spark data -d sales -c zstd -f orc
Review Comment:
I get an error when I do the following according to the README of bench.
Since `bench/core`, `bench/hive` are packaged with `maven-assembly-plugin`, the jar contains uber, and bench/spark is packaged with `maven-shade-plugin`, the generated jar file name doesn't contain `shaded` in case `shadedArtifactAttached= false`, the generated jar filename does not contain `shaded`.
```bash
cd java
./mvnw clean package -Pbenchmark -DskipTests
cd bench
java -jar spark/target/orc-benchmarks-spark-*.jar spark data
```
```
Error: Invalid or corrupt jarfile spark/target/orc-benchmarks-spark-2.1.0-SNAPSHOT-nohive.jar
```
bench/spark/target
```
-rw-r--r--@ 1 csy staff 2.1M 1 17 12:03 orc-benchmarks-spark-2.1.0-SNAPSHOT-nohive.jar
-rw-r--r-- 1 csy staff 1.8M 1 17 12:03 orc-benchmarks-spark-2.1.0-SNAPSHOT-shaded-protobuf.jar
-rw-r--r-- 1 csy staff 39K 1 17 12:03 orc-benchmarks-spark-2.1.0-SNAPSHOT-sources.jar
-rw-r--r-- 1 csy staff 13K 1 17 12:03 orc-benchmarks-spark-2.1.0-SNAPSHOT-test-sources.jar
-rw-r--r-- 1 csy staff 13K 1 17 12:04 orc-benchmarks-spark-2.1.0-SNAPSHOT-tests.jar
-rw-r--r--@ 1 csy staff 217M 1 17 12:04 orc-benchmarks-spark-2.1.0-SNAPSHOT.jar
-rw-r--r-- 1 csy staff 64K 1 17 12:03 original-orc-benchmarks-spark-2.1.0-SNAPSHOT.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1749:
URL: https://github.com/apache/orc/pull/1749#discussion_r1454621856
##########
java/CMakeLists.txt:
##########
@@ -97,7 +97,7 @@ add_test(
add_test(
NAME java-bench-spark-test
- COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar spark data -d sales -c zstd -f orc
+ COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}-shaded.jar spark data -d sales -c zstd -f orc
Review Comment:
@cxzl25 , it sounds like you misunderstood the error message. Technically, your error message is irrelevant with this line because this code is the following which uses exactly `${ORC_VERSION}.jar`.
```
bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar
```
Please revert this line and leave the documentation revision only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1749:
URL: https://github.com/apache/orc/pull/1749#discussion_r1454707503
##########
java/CMakeLists.txt:
##########
@@ -97,7 +97,7 @@ add_test(
add_test(
NAME java-bench-spark-test
- COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar spark data -d sales -c zstd -f orc
+ COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}-shaded.jar spark data -d sales -c zstd -f orc
Review Comment:
Done.
My original intention is to execute it in * way, so that I don’t need to care about the variable `ORC_VERSION`, just like other benchmarks (` java -jar hive/target/orc-benchmarks-hive-*-uber.jar`) .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "mystic-lama (via GitHub)" <gi...@apache.org>.
mystic-lama commented on PR #1749:
URL: https://github.com/apache/orc/pull/1749#issuecomment-1891297060
Can we be more descriptive in description as to what we are trying to do? This shall help later.
I am able to reproduce this, however we also need to find out why we are getting the corrupt jar error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1749:
URL: https://github.com/apache/orc/pull/1749#discussion_r1453652434
##########
java/CMakeLists.txt:
##########
@@ -97,7 +97,7 @@ add_test(
add_test(
NAME java-bench-spark-test
- COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar spark data -d sales -c zstd -f orc
+ COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}-shaded.jar spark data -d sales -c zstd -f orc
Review Comment:
You described `-nohive `in the PR description like the following, but we don't use `-nohive` at line 100.
May I ask why you did propose to change a working code by giving an irrelevant example?
```
Error: Invalid or corrupt jarfile spark/target/orc-benchmarks-spark-2.1.0-SNAPSHOT-nohive.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1749:
URL: https://github.com/apache/orc/pull/1749#issuecomment-1891299166
> however we also need to find out why we are getting the corrupt jar error.
https://github.com/apache/orc/blob/ffb0c87c94c8e1b5e0a7d722586585a1e1c23cfa/java/pom.xml#L536-L549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1749:
URL: https://github.com/apache/orc/pull/1749#discussion_r1453647830
##########
java/CMakeLists.txt:
##########
@@ -97,7 +97,7 @@ add_test(
add_test(
NAME java-bench-spark-test
- COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar spark data -d sales -c zstd -f orc
+ COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}-shaded.jar spark data -d sales -c zstd -f orc
Review Comment:
Sorry, but this is wrong, @cxzl25 . Apache ORC community should use non-shaded version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1749:
URL: https://github.com/apache/orc/pull/1749#discussion_r1454621856
##########
java/CMakeLists.txt:
##########
@@ -97,7 +97,7 @@ add_test(
add_test(
NAME java-bench-spark-test
- COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar spark data -d sales -c zstd -f orc
+ COMMAND java ${ADD_OPENS} ${JAVA_NIO} -jar bench/spark/orc-benchmarks-spark-${ORC_VERSION}-shaded.jar spark data -d sales -c zstd -f orc
Review Comment:
@cxzl25 , it sounds like you misunderstood the error message. 😄 Technically, your error message is irrelevant with this line because this code is the following which uses exactly `${ORC_VERSION}.jar`.
```
bench/spark/orc-benchmarks-spark-${ORC_VERSION}.jar
```
Please revert this line and leave the documentation revision only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1587: Fix usage command of SparkBenchmark document [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1749: ORC-1587: Fix usage command of SparkBenchmark document
URL: https://github.com/apache/orc/pull/1749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org