[jira] [Commented] (HUDI-1204) NoClassDefFoundError with AbstractSyncTool while running HoodieTestSuiteJob

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180850#comment-17180850 ] 

sivabalan narayanan commented on HUDI-1204:
-------------------------------------------

Steps I followed. 

Did some changes to some config files.

 

*diff --git a/docker/demo/config/test-suite/test-source.properties b/docker/demo/config/test-suite/test-source.properties*

*index 397f871c..4cd06c3b 100644*

*--- a/docker/demo/config/test-suite/test-source.properties*

*+++ b/docker/demo/config/test-suite/test-source.properties*

@@ -17,11 +17,11 @@ hoodie.datasource.write.recordkey.field=_row_key

 hoodie.deltastreamer.source.dfs.root=/user/hive/warehouse/hudi-bench/input

 hoodie.datasource.write.keygenerator.class=org.apache.hudi.ComplexKeyGenerator

 hoodie.datasource.write.partitionpath.field=timestamp

-hoodie.deltastreamer.schemaprovider.source.schema.file=/var/hoodie/ws/docker/demo/config/bench/source.avsc

+hoodie.deltastreamer.schemaprovider.source.schema.file=/var/hoodie/ws/docker/demo/config/test-suite/source.avsc

 hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://hiveserver:10000/

 hoodie.datasource.hive_sync.database=testdb

 hoodie.datasource.hive_sync.table=test_table

 hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPartitionedExtractor

 hoodie.datasource.hive_sync.assume_date_partitioning=true

 hoodie.datasource.write.keytranslator.class=org.apache.hudi.DayBasedPartitionPathKeyTranslator

-hoodie.deltastreamer.schemaprovider.target.schema.file=/var/hoodie/ws/docker/demo/config/bench/source.avsc

\ No newline at end of file

+hoodie.deltastreamer.schemaprovider.target.schema.file=/var/hoodie/ws/docker/demo/config/test-suite/source.avsc

 

Copied following files to adhoc-2:/opt/

hudi-integ-test-bundle-*.jar

test-source.properties

complex-dag-cow.yaml

source.avsc

 

After that, ran the command given in description. 

 

> NoClassDefFoundError with AbstractSyncTool while running HoodieTestSuiteJob
> ---------------------------------------------------------------------------
>
>                 Key: HUDI-1204
>                 URL: https://issues.apache.org/jira/browse/HUDI-1204
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: Testing
>    Affects Versions: 0.6.1
>            Reporter: sivabalan narayanan
>            Assignee: Nishith Agarwal
>            Priority: Major
>
> I was trying to run HoodieTestSuiteJob in my local docker set up and ran into dep issue.
>  
> spark-submit --master local --class org.apache.hudi.integ.testsuite.HoodieTestSuiteJob --packages com.databricks:spark-avro_2.11:4.0.0 /opt/hudi-integ-test-bundle-0.6.0-rc1.jar  --source-ordering-field timestamp    --target-base-path /user/hive/warehouse/hudi-test-suite/output    --input-base-path /user/hive/warehouse/hudi-test-suite/input    --target-table test_table    --props [file:///opt/test-source.properties]    --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider    --source-class org.apache.hudi.utilities.sources.AvroDFSSource    --input-file-size 12582912  --workload-yaml-path /var/hoodie/ws/docker/demo/config/test-suite/complex-dag-cow.yaml --table-type COPY_ON_WRITE    --workload-generator-classname yaml
>  
> ```
> 20/08/19 21:42:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hudi/sync/common/AbstractSyncTool
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$Config.<init>(HoodieDeltaStreamer.java:279)
> at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob$HoodieTestSuiteConfig.<init>(HoodieTestSuiteJob.java:153)
> at org.apache.hudi.integ.testsuite.HoodieTestSuiteJob.main(HoodieTestSuiteJob.java:114)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: org.apache.hudi.sync.common.AbstractSyncTool
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 26 more
> ```
> I tried adding hudi-sync-common as dep to hudi-utilities, but didn't fix the issue. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)