You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/21 06:57:24 UTC
[GitHub] [hudi] hanson2021 opened a new issue #3838: [SUPPORT]Can Hudi support more hive version?
hanson2021 opened a new issue #3838:
URL: https://github.com/apache/hudi/issues/3838
Hadoop: 3.0.0-cdh6.3.2
Hive：2.1.1-cdh6.3.2
Flink: 1.13.1
Scala: 2.11
(1) set hudi/pom.xml hive.version=2.1.1 to recompile like this：
mvn clean install -DskipTests -DskipITs -Dcheckstyle.skip=true -Drat.skip=true -Dscala-2.11 -Dhive.version=2.1.1 -Pflink-bundle-shade-hive2 -Dflink.version=1.13.1
Error info as following:
```
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-hadoop-mr: Compilation failure
[ERROR] /data1/download/apachehudi/hudi-0.10.0/tmp/hudi/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java:[32,37] 找不到符号
[ERROR] 符号: 类 StringInternUtils
[ERROR] 位置: 程序包 org.apache.hadoop.hive.common
[ERROR]
[ERROR] -> [Help 1]
```
(2) After use default pom.xml to recompile ,when run flink sql on yarn with hudi、hive、flink, Error info as following:
```
2021-10-21 10:56:00,215 INFO hive.metastore [] - Closed a connection to metastore, current connections: 0
2021-10-21 10:56:00,215 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [sync hive metadata for instant 20211021105600] error
org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing unknown
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:120) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:305) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if table exists unknown_ro
at org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:214) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1443) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.tableExists(SessionHiveMetaStoreClient.java:266) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at org.apache.hudi.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2303) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:212) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
2021-10-21 10:56:01,034 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 192 for job aa1c0e656a89bab1cd28f729d3356fba (56712 bytes in 1982 ms).
2021-10-21 10:56:01,037 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 193 (type=CHECKPOINT) @ 1634784961035 for job aa1c0e656a89bab1cd28f729d3356fba.
2021-10-21 10:56:01,080 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Commit instant [20211021105600] success!
2021-10-21 10:56:02,041 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Create instant [20211021105602] for table [hudi_lcy_test2] with type [MERGE_ON_READ]
2021-10-21 10:56:02,041 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [commits the instant 20211021105600] success!
2021-10-21 10:56:02,041 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [taking checkpoint 193] success!
2021-10-21 10:56:02,053 INFO hive.metastore [] - Trying to connect to metastore with URI thrift://sh1-bi-thdwcdh-hm02:9083
2021-10-21 10:56:02,053 INFO hive.metastore [] - Opened a connection to metastore, current connections: 1
2021-10-21 10:56:02,053 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [handle write metadata event for instant 20211021105602] success!
2021-10-21 10:56:02,054 INFO hive.metastore [] - Connected to metastore.
2021-10-21 10:56:02,146 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [handle write metadata event for instant 20211021105602] success!
2021-10-21 10:56:02,209 INFO hive.metastore [] - Closed a connection to metastore, current connections: 0
2021-10-21 10:56:02,210 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [sync hive metadata for instant 20211021105602] error
org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing unknown
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:120) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:305) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:67) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if table exists unknown_ro
at org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:214) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[flink-sql-connector-hive-2.2.0_2.11-1.13.0.jar:1.13.0]
at org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1443) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.tableExists(SessionHiveMetaStoreClient.java:266) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at org.apache.hudi.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at org.apache.hudi.org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2303) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at com.sun.proxy.$Proxy69.tableExists(Unknown Source) ~[?:?]
at org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:212) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:167) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:135) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:117) ~[hudi-flink-bundle_2.11-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 5 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Guanpx commented on issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
Guanpx commented on issue #3838:
URL: https://github.com/apache/hudi/issues/3838#issuecomment-1072272543
hi there, modify this version:https://github.com/apache/hudi/blob/master/packaging/hudi-flink-bundle/pom.xml#L39 , change hive version 2.3.1 to 2.1.1, then sync hive will be ok~ (flink1.14 + master(0.11))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hanson2021 commented on issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
hanson2021 commented on issue #3838:
URL: https://github.com/apache/hudi/issues/3838#issuecomment-949420628
> @codope do we have a list of supported hive versions? guess `2.1.1-cdh6.3.2` is not certified to work with hudi?
```
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
```
Invalid method name 'get_table_req' is supported at hive version > 2.3.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3838:
URL: https://github.com/apache/hudi/issues/3838#issuecomment-968402240
solution was suggested above. closing due to inactive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3838:
URL: https://github.com/apache/hudi/issues/3838#issuecomment-949244404
@codope do we have a list of supported hive versions? guess `2.1.1-cdh6.3.2` is not certified to work with hudi?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Guanpx commented on issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
Guanpx commented on issue #3838:
URL: https://github.com/apache/hudi/issues/3838#issuecomment-1072272543
hi there, modify this version:https://github.com/apache/hudi/blob/master/packaging/hudi-flink-bundle/pom.xml#L39 , change hive version 2.3.1 to 2.1.1, then sync hive will be ok~ (flink1.14 + master(0.11))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3838:
URL: https://github.com/apache/hudi/issues/3838#issuecomment-949440451
> guess 2.1.1-cdh6.3.2 is not certified to work with hudi?
No. We have only ceertified 2.3.1, 2.3.3, 2.3.7. Certification for Hive 3 is in progress.
`ThriftHiveMetastore` does not have `get_table_req` in [version 2.1.1](https://github.com/apache/hive/blob/rel/release-2.1.1/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h). As @hanson2021 rightly pointed out that it was added in 2.3.0. So, the solution here is to upgrade to 2.3.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope edited a comment on issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
codope edited a comment on issue #3838:
URL: https://github.com/apache/hudi/issues/3838#issuecomment-949440451
> guess 2.1.1-cdh6.3.2 is not certified to work with hudi?
No. We have only ceertified 2.3.1, 2.3.3, 2.3.7. Certification for Hive 3 is in progress.
`ThriftHiveMetastore` does not have `get_table_req` in [version 2.1.1](https://github.com/apache/hive/blob/rel/release-2.1.1/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h). As @hanson2021 rightly pointed out that it was added in 2.3.0. So, the solution here is to upgrade to 2.3.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #3838: [SUPPORT]Can Hudi support more hive version?
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3838:
URL: https://github.com/apache/hudi/issues/3838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org