You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/07/10 11:46:45 UTC

[GitHub] [hudi] RajasekarSribalan opened a new issue #1814: [SUPPORT] Needed Hudi comapction job spark command to submit it manually

RajasekarSribalan opened a new issue #1814:
URL: https://github.com/apache/hudi/issues/1814


   Hi,
   
   We are trying to run compaction job for MOR tables via hudi cli. Unfortunately , as we use cloudera cluster we have both Spark 1.6 and Spark 2.2.0 in our cluster. 
   
   So,
   
   spark-submit  - 1.6
   spark2-submit - 2.2.0
   
   Even though we provide the SPARK_HOME(2.2.0), the compaction submits the job in Spark 1.6 but we need the spark compaction to be submitted  Spark2-submit i.e., 2.2.0. I understand ,we use Apache's SparkLauncher internally so it submits the jobs with spark-submit but not with spark2-submit. 
   
   Could you please share the manual spark submit command to run compaction job for MOR tables so that we can schedule it manually?  
   
   with 1.6 version, compaction job is failing with below error.
   
   ERROR utilities.HoodieCompactor: java.lang.NoSuchMethodError: org.eclipse.jetty.util.thread.QueuedThreadPool.<init>(III)V
   
   Please provide your support. and do please let us know how we can do compaction for MOR tables. We get 100million records every day for single table with 50:50 insert to upsert ratio.
   
   Pls help @vinothchandar @bhasudha 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] RajasekarSribalan closed issue #1814: [SUPPORT] Needed Hudi compaction job spark command to submit it manually

Posted by GitBox <gi...@apache.org>.
RajasekarSribalan closed issue #1814:
URL: https://github.com/apache/hudi/issues/1814


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org