You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/04/15 22:43:28 UTC
[GitHub] [hudi] nevgin opened a new issue #2832: [SUPPORT]
nevgin opened a new issue #2832:
URL: https://github.com/apache/hudi/issues/2832
I have installed vanilla versions of hive and spark.
Put the jar hoodie spark bundle in the spark. Put hudi-hadoop-mr-bundle-x.y.z-SNAPSHOT.jar in aux hive dir and to classpath hadoop on all datanodes.
When a query is executed in hive with the MP engine, the queries are executed. Requests also be executed from spark.
However, when running a query to Hive with Engine Spark, an error is thrown
Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.l
ang.NoClassDefFoundError: org / apache / hadoop / hive / ql / io / parquet / MapredParquetInputFormat
**Expected behavior**
Executed query
**Environment Description**
* Hudi version :
0.7.0
* Spark version :
2.4.7
* Hive version :
2.3.8
* Hadoop version :
2.7.3
* Storage (HDFS/S3/GCS..) :
HDFS
* Running on Docker? (yes/no) :
NO
**Additional context**
Add any other context about the problem here.
**Stacktrace**
java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nevgin commented on issue #2832: [SUPPORT] Hive on Spark dont work
Posted by GitBox <gi...@apache.org>.
nevgin commented on issue #2832:
URL: https://github.com/apache/hudi/issues/2832#issuecomment-823117798
Directly from the spark, queries are being handled wonderfully.
From spark for hive, according to the documentation, I removed the hive*.jar libraries. If you do not delete the hive does not work with the spark engine.
my guess is that the spark needs to be built with my version of the hive - 2.3.8.
This is true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2832: [SUPPORT] Hive on Spark dont work
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2832:
URL: https://github.com/apache/hudi/issues/2832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2832: [SUPPORT] Hive on Spark dont work
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2832:
URL: https://github.com/apache/hudi/issues/2832#issuecomment-826558842
@nevgin Yes, this is true. Spark 2.x still uses Hive 1.x (https://github.com/apache/spark/blob/v2.4.6/pom.xml#L129) whereas spark 3.x uses Hive 2.x (https://github.com/apache/spark/blob/master/pom.xml#L131). Since Hudi support Hive 2.x+, this workaround is needed for spark 2.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2832: [SUPPORT] Hive on Spark dont work
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2832:
URL: https://github.com/apache/hudi/issues/2832#issuecomment-854408139
@nevgin Closing this issue due to inactivity. Please re-open if you need further assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2832: [SUPPORT] Hive on Spark dont work
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2832:
URL: https://github.com/apache/hudi/issues/2832#issuecomment-822552819
can you please give us the command you used to launch spark and ran your commands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2832: [SUPPORT] Hive on Spark dont work
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2832:
URL: https://github.com/apache/hudi/issues/2832#issuecomment-854408139
@nevgin Closing this issue due to inactivity. Please re-open if you need further assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2832: [SUPPORT] Hive on Spark dont work
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2832:
URL: https://github.com/apache/hudi/issues/2832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org