You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Anton Zuyeu <an...@gmail.com> on 2020/06/15 05:27:52 UTC

Hudi + AWS GLue NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.

Hi,

When running Hudi  on AWS EMR cluster with enabled Glue support I am
getting the following exception:
NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.
By searching the internet I discovered this page
https://forums.aws.amazon.com/thread.jspa?messageID=930176 that suggests
adding httpclient-4.5.9.jar to list of --jars  when running spark-submitt.
Suggestion helped. Now I am trying to validate Hudi 0.5.3 that is not part
of the latest EMR release. To do so I built supper jar (uber jar) that
includes my spark job code along with Hudi 0.5.3 jar, when attempting to
run my super jar on EMR cluster with Glue support I am getting the
same NoSuchMethodError
exception as I used to. adding httpclient-4.5.9.jar to list of --jars this
time doesn't help

Any help will be appreciated,
Anton

Re: Hudi + AWS GLue NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.

Posted by "Mehrotra, Udit" <ud...@amazon.com.INVALID>.
Hi Anton,

This issue seems specific to AWS EMR, and it would be best for you to open a support ticket with the service if you are able to reproduce the issue with Hudi version that is supported on the EMR release.

On a parallel note, if you really want to build Hudi 0.5.3 you do not have to build an uber jar that includes your spark code as well as hudi. Hudi needs to be built separately and you will have to replace the bundle jars under /usr/lib/hudi,  and your code should be a separate jar that you pass to spark-submit. The way you are currently doing, you will end up with duplicate Hudi classes in spark's classpath and it can cause runtime issues.

Thanks,
Udit

On 6/14/20, 10:29 PM, "Anton Zuyeu" <an...@gmail.com> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
    
    
    
    Hi,
    
    When running Hudi  on AWS EMR cluster with enabled Glue support I am
    getting the following exception:
    NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.
    By searching the internet I discovered this page
    https://forums.aws.amazon.com/thread.jspa?messageID=930176 that suggests
    adding httpclient-4.5.9.jar to list of --jars  when running spark-submitt.
    Suggestion helped. Now I am trying to validate Hudi 0.5.3 that is not part
    of the latest EMR release. To do so I built supper jar (uber jar) that
    includes my spark job code along with Hudi 0.5.3 jar, when attempting to
    run my super jar on EMR cluster with Glue support I am getting the
    same NoSuchMethodError
    exception as I used to. adding httpclient-4.5.9.jar to list of --jars this
    time doesn't help
    
    Any help will be appreciated,
    Anton