You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2022/09/20 15:25:00 UTC
[jira] [Updated] (HUDI-4885) docker demo fails w/ ClassNotFound w/ LogicalType in latest master
[ https://issues.apache.org/jira/browse/HUDI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-4885:
--------------------------------------
Description:
docker demo fails during hive-sync w/ latest master.
also, some of the env variables are not applied. for eg, HUDI_UTILITIES_BUNDLE was not set.
command used:
{code:java}
/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh --jdbc-url jdbc:hive2://hiveserver:10000 --user hive --pass hive --partitioned-by dt --base-path /user/hive/warehouse/stock_ticks_cow --database default --table stock_ticks_cow {code}
output:
{code:java}
2022-09-20 14:24:39,122 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(179)) - Trying to sync hoodie table stock_ticks_cow with base path /user/hive/warehouse/stock_ticks_cow of type COPY_ON_WRITE
2022-09-20 14:24:39,758 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(439)) - Reading schema from /user/hive/warehouse/stock_ticks_cow/2018/08/31/b4a7076c-30e6-4320-bb04-be47246b6646-0_0-29-29_20220920142351042.parquet
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2022-09-20 14:24:40,432 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 0
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
at org.apache.hudi.common.table.TableSchemaResolver.convertParquetSchemaToAvro(TableSchemaResolver.java:288)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:121)
at org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:566)
at org.apache.hudi.util.Lazy.get(Lazy.java:53)
at org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromLatestCommitMetadata(TableSchemaResolver.java:225)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaInternal(TableSchemaResolver.java:193)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:142)
at org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchema(TableSchemaResolver.java:173)
at org.apache.hudi.sync.common.HoodieSyncClient.getStorageSchema(HoodieSyncClient.java:103)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:206)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:153)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:358)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.LogicalType
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more {code}
tried the spark-submit command directly and it succeeded.
{code:java}
spark-submit --class org.apache.hudi.hive.HiveSyncTool /var/hoodie/ws/packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.13.0-SNAPSHOT.jar --database default --table stock_ticks_cow --base-path /user/hive/warehouse/stock_ticks_cow --base-file-format PARQUET --user hive --pass hive --jdbc-url jdbc:hive2://hiveserver:10000 --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor --partitioned-by dt {code}
was:
docker demo fails during hive-sync w/ latest master.
also, some of the env variables are not applied. for eg, HUDI_UTILITIES_BUNDLE was not set.
command used:
{code:java}
/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh --jdbc-url jdbc:hive2://hiveserver:10000 --user hive --pass hive --partitioned-by dt --base-path /user/hive/warehouse/stock_ticks_cow --database default --table stock_ticks_cow {code}
output:
{code:java}
2022-09-20 14:24:39,122 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(179)) - Trying to sync hoodie table stock_ticks_cow with base path /user/hive/warehouse/stock_ticks_cow of type COPY_ON_WRITE
2022-09-20 14:24:39,758 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(439)) - Reading schema from /user/hive/warehouse/stock_ticks_cow/2018/08/31/b4a7076c-30e6-4320-bb04-be47246b6646-0_0-29-29_20220920142351042.parquet
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2022-09-20 14:24:40,432 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 0
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
at org.apache.hudi.common.table.TableSchemaResolver.convertParquetSchemaToAvro(TableSchemaResolver.java:288)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:121)
at org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:566)
at org.apache.hudi.util.Lazy.get(Lazy.java:53)
at org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromLatestCommitMetadata(TableSchemaResolver.java:225)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaInternal(TableSchemaResolver.java:193)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:142)
at org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchema(TableSchemaResolver.java:173)
at org.apache.hudi.sync.common.HoodieSyncClient.getStorageSchema(HoodieSyncClient.java:103)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:206)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:153)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:358)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.LogicalType
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more {code}
> docker demo fails w/ ClassNotFound w/ LogicalType in latest master
> ------------------------------------------------------------------
>
> Key: HUDI-4885
> URL: https://issues.apache.org/jira/browse/HUDI-4885
> Project: Apache Hudi
> Issue Type: Bug
> Components: dev-experience
> Reporter: sivabalan narayanan
> Priority: Major
>
> docker demo fails during hive-sync w/ latest master.
> also, some of the env variables are not applied. for eg, HUDI_UTILITIES_BUNDLE was not set.
> command used:
> {code:java}
> /var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh --jdbc-url jdbc:hive2://hiveserver:10000 --user hive --pass hive --partitioned-by dt --base-path /user/hive/warehouse/stock_ticks_cow --database default --table stock_ticks_cow {code}
>
> output:
> {code:java}
> 2022-09-20 14:24:39,122 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(179)) - Trying to sync hoodie table stock_ticks_cow with base path /user/hive/warehouse/stock_ticks_cow of type COPY_ON_WRITE
> 2022-09-20 14:24:39,758 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(439)) - Reading schema from /user/hive/warehouse/stock_ticks_cow/2018/08/31/b4a7076c-30e6-4320-bb04-be47246b6646-0_0-29-29_20220920142351042.parquet
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> 2022-09-20 14:24:40,432 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 0
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
> at org.apache.hudi.common.table.TableSchemaResolver.convertParquetSchemaToAvro(TableSchemaResolver.java:288)
> at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:121)
> at org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:566)
> at org.apache.hudi.util.Lazy.get(Lazy.java:53)
> at org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromLatestCommitMetadata(TableSchemaResolver.java:225)
> at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaInternal(TableSchemaResolver.java:193)
> at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:142)
> at org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchema(TableSchemaResolver.java:173)
> at org.apache.hudi.sync.common.HoodieSyncClient.getStorageSchema(HoodieSyncClient.java:103)
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:206)
> at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:153)
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
> at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:358)
> Caused by: java.lang.ClassNotFoundException: org.apache.avro.LogicalType
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more {code}
>
>
> tried the spark-submit command directly and it succeeded.
> {code:java}
> spark-submit --class org.apache.hudi.hive.HiveSyncTool /var/hoodie/ws/packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-0.13.0-SNAPSHOT.jar --database default --table stock_ticks_cow --base-path /user/hive/warehouse/stock_ticks_cow --base-file-format PARQUET --user hive --pass hive --jdbc-url jdbc:hive2://hiveserver:10000 --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor --partitioned-by dt {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)