You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/18 12:04:31 UTC

[GitHub] [hudi] Aload opened a new issue, #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Aload opened a new issue, #5346:
URL: https://github.com/apache/hudi/issues/5346

   hi , When I use Flink to consume Kakfa and write hudi, I configure the hive synchronization operation. The same problem occurred during startup .
   When I looked through the hivesyncContext.java source code, I noticed that the package imported under the source code was very strange. But this is normal when I don't synchronize hive.
   
   **version: hudi 0.10.1 flink 1.13.1 scala 2.12.10**
   eg:
   private val sinkSql: String = SinkSql.apply(sinkTableName, dataType.getLogicalType.asInstanceOf[RowType]) .option(FlinkOptions.PATH, hoodiePath.concat(sinkDbName).concat("/").concat(sinkTableName).concat("/")) .option(FlinkOptions.PRECOMBINE_FIELD, "receiveTime") .option(FlinkOptions.RECORD_KEY_FIELD, "tenantId,pointId,sn,collectTime") .option(FlinkOptions.READ_AS_STREAMING, true) .option(FlinkOptions.READ_START_COMMIT, "earliest") .option(FlinkOptions.OPERATION, WriteOperationType.INSERT) .option(FlinkOptions.WRITE_BULK_INSERT_SHUFFLE_BY_PARTITION, true) .option(FlinkOptions.TABLE_TYPE, HoodieTableType.COPY_ON_WRITE) .option(FlinkOptions.BUCKET_ASSIGN_TASKS, 10) .option(FlinkOptions.COMPACTION_ASYNC_ENABLED, true) .option(FlinkOptions.COMPACTION_DELTA_COMMITS, 1) .option(FlinkOptions.READ_STREAMING_CHECK_INTERVAL, 10) // .option(FlinkOptions.INSERT_CLUSTER, true) .option(FlinkOptions.RETRY_TIMES, 5) .option(FlinkOptions.INSERT_CLUSTER, true) .option(FlinkOptions.WRITE_TASKS, 10) 
 .option(FlinkOptions.HIVE_SYNC_ENABLED, true) .option(FlinkOptions.HIVE_SYNC_AUTO_CREATE_DB, true) .option(FlinkOptions.HIVE_SYNC_DB, "ods") .option(FlinkOptions.HIVE_SYNC_TABLE, sinkTableName) .option(FlinkOptions.HIVE_SYNC_MODE, "hms") .option(FlinkOptions.HIVE_SYNC_METASTORE_URIS, "thrift://dev32:9083") .option(FlinkOptions.HIVE_SYNC_JDBC_URL, "jdbc:hive2://dev32:10000") .option(FlinkOptions.HIVE_STYLE_PARTITIONING, true) .option(FlinkOptions.HIVE_SYNC_SUPPORT_TIMESTAMP, true) .partitionField("tenantId", "fmy", "fmm", "fmd") // .option(FlinkOptions.INDEX_GLOBAL_ENABLED, true) .end
   
   **Environment Description**
   
   * Hudi version : 0.10.1
   
   * Flink version : 1.13.1
    
   *  Hive version : 2.3.7
   
   * Hadoop version : 2.7.3<
   
   * Storage (HDFS/S3/GCS..) :HDFS
   
   * Running on Docker? (yes/no) :no
   
   ![image](https://user-images.githubusercontent.com/13082598/163805664-151284ee-e7d8-4d37-be8a-2cee5ad4ebe3.png)
   ![image](https://user-images.githubusercontent.com/13082598/163805705-27739eb1-14da-4f7d-b152-9fb877c32618.png)
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Aload closed issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Posted by GitBox <gi...@apache.org>.
Aload closed issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf
URL: https://github.com/apache/hudi/issues/5346


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Posted by GitBox <gi...@apache.org>.
codope commented on issue #5346:
URL: https://github.com/apache/hudi/issues/5346#issuecomment-1104000813

   @Aload you see that import in the decompiled class because hive dependency is shaded in hudi-flink-bundle. If you look at the [source code](https://github.com/apache/hudi/blob/release-0.10.1/hudi-flink/src/main/java/org/apache/hudi/sink/utils/HiveSyncContext.java) then the imports are fine.
   
   I've not run hive sync from IDE. My guess is this happens in IDE and you may have to change your settings to include dependencies in provided scope. Can you please try to run hive sync tool from command line?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] yihua commented on issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5346:
URL: https://github.com/apache/hudi/issues/5346#issuecomment-1111749145

   @danny0405 do you have any idea on why the `HiveConf` class is not found from the Flink bundle?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] puchangchun commented on issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Posted by GitBox <gi...@apache.org>.
puchangchun commented on issue #5346:
URL: https://github.com/apache/hudi/issues/5346#issuecomment-1101915682

   ```
           <dependency>
               <groupId>org.apache.hive</groupId>
               <artifactId>hive-exec</artifactId>
               <version>2.3.9</version>
           </dependency>
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5346:
URL: https://github.com/apache/hudi/issues/5346#issuecomment-1111844591

   You need to package the jar under path `packaging/hudi-flink-bundle`, if you package the jar under root path, the shade pattern for Hive is overridden, this is a bug i guess.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Aload commented on issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Posted by GitBox <gi...@apache.org>.
Aload commented on issue #5346:
URL: https://github.com/apache/hudi/issues/5346#issuecomment-1104638847

   > 没有从 ID
   
   OK  
   
   > ```
   >         <dependency>
   >             <groupId>org.apache.hive</groupId>
   >             <artifactId>hive-exec</artifactId>
   >             <version>2.3.9</version>
   >         </dependency>
   > ```
   
   get  ,I've tried it your way, but it keeps happening
   ![image](https://user-images.githubusercontent.com/13082598/164358619-630f01ca-929b-4afd-916d-dab9b3aecea9.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] Aload commented on issue #5346: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hive/conf/HiveConf

Posted by GitBox <gi...@apache.org>.
Aload commented on issue #5346:
URL: https://github.com/apache/hudi/issues/5346#issuecomment-1104645247

   > @Aload您会在反编译的类中看到该导入，因为 hive 依赖项在 hudi-flink-bundle 中是阴影的。如果您查看[源代码](https://github.com/apache/hudi/blob/release-0.10.1/hudi-flink/src/main/java/org/apache/hudi/sink/utils/HiveSyncContext.java)，则导入很好。
   > 
   > 我没有从 IDE 运行配置单元同步。我的猜测是这发生在 IDE 中，您可能必须更改设置以在提供的范围内包含依赖项。您可以尝试从命令行运行配置单元同步工具吗？
   
   hi ,I also introduced the hud-flink-bundle inside the IDE, but the same problem occurred.  
   ![image](https://user-images.githubusercontent.com/13082598/164360183-facf10d1-e5eb-486e-953c-e982b2753401.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org