You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/31 09:27:06 UTC
[GitHub] [hudi] niloo-sh opened a new issue #3572: compatble version of hudi, hive and hadoop
niloo-sh opened a new issue #3572:
URL: https://github.com/apache/hudi/issues/3572
Dear Hudi team member,
These days we want to use Hudi as data lake on HDFS (3.1.1) and synchronize Hudi table with Hive(3.1.0).
Contrary to what has been said in FAQ Hudi table is created and updated in HDFS with Datasource API, but I get errors in synchronizing it with Hive.
Has there been a change in the compatible versions?
We use Scala 2.11.7 and Spark 2.4.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh edited a comment on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh edited a comment on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-912928844
@vinothchandar you mean I can not use Hudi with hdfs(3.1.1) and hive(3.1.0)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-997308285
@niloo-sh : can you try using run_hive_sync tool which is the most common way : https://hudi.apache.org/docs/docker_demo#step-3-sync-with-hive
and let us know how it goes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-1020591206
Closing this due to inactivity. Please feel free to re-open if you need assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-928903882
@vinothchandar thank you, I added dependency to build.sbt file and I created fat jar and run it with spark-submit. now I get exception:"org/json/jsonexception", its correct, I use hudi-spark-bundle-0.9.0 and it does not have json. I download json-20210307.jar and add it to "spark.driver.extraClassPath" and "spark.executor.extraclasspath", but it does not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-919733664
@nsivabalan yes, I started with quick start and continue with Datasource Writer, but I gettin into trouble when I want to sync Hudi table with hive(3.1.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] borasy edited a comment on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
borasy edited a comment on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-927743362
`hudi-spark` is different frmm hudi-spark-bundle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-928903882
@vinothchandar thank you, I added dependency to build.sbt file and I created fat jar and run it with spark-submit. now I get exception:"org/json/jsonexception", its correct, I use hudi-spark-bundle-0.9.0 and it does not have json. I download json-20210307.jar and add it to "spark.driver.extraClassPath" and "spark.executor.extraclasspath", but it does not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh removed a comment on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh removed a comment on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-912929026
@vinothchandar you mean I can not use, hdfs(3.1.1) and hive(3.1.0)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-912928844
@vinothchandar you mean I can not use, hdfs(3.1.1) and hive(3.1.0)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3572:
URL: https://github.com/apache/hudi/issues/3572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-926266895
@niloo-sh thats because hive is a provided dependency. you need to add one of the hudi bundles in the module class path to make it work from IDE>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] borasy commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
borasy commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-924802949
i also cannot sync hudi table to hive.
-----
hudi 0.9
hive 3.1.2
spark 3.0.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-997308285
@niloo-sh : can you try using run_hive_sync tool which is the most common way : https://hudi.apache.org/docs/docker_demo#step-3-sync-with-hive
and let us know how it goes.
you can also read more about metastore sync [here](https://hudi.apache.org/docs/syncing_metastore).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-997341300
also, can you give us full stacktrace and configs you are running with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-1002291208
@niloo-sh : do you have any updates for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-926267139
Also note that you probably should depend on `hudi-spark` and not hudi-spark-bundle to write programs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-910388899
there has not been any changes in compatibility. what errors are you seeing, could you share stack trace?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-926267261
@borasy are you also trying via IDE>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3572:
URL: https://github.com/apache/hudi/issues/3572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-919727432
@niloo-sh : did you try quick start utils and running into issues as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-912929026
@vinothchandar you mean I can not use, hdfs(3.1.1) and hive(3.1.0)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-1020591206
Closing this due to inactivity. Please feel free to re-open if you need assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-997341166
@niloo-sh : I see you have still using hudi 0.6.0. Can you try one of the latest version like 0.9.0 or higher. we don't think hive 3 has been tested with older versions of hudi.
Also, would be easier to run hive sync tool from terminal rather than IDE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] borasy commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
borasy commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-927743362
`hudi-spark` is different form hudi-spark-bundle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] niloo-sh commented on issue #3572: compatble version of hudi, hive and hadoop
Posted by GitBox <gi...@apache.org>.
niloo-sh commented on issue #3572:
URL: https://github.com/apache/hudi/issues/3572#issuecomment-922326982
@vinothchandar I use intellij IDE, add libraryDependencies += "org.apache.hudi" %% "hudi-spark-bundle" % "0.6.0" in build.sbt and get this error :
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.hudi.HoodieSparkSqlWriter$.org$apache$hudi$HoodieSparkSqlWriter$$syncHive(HoodieSparkSqlWriter.scala:319)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org