running Hudi in AWS Glue Spark

["Sanchez, Jorge" <jo...@merck.com.INVALID>]

Hello,

Did anybody tried to run Hudi within AWS Glue job, I searched the JIRA issues but did not find anybody mentioning that.


Thanks,

Jorge
Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (2000 Galloping Hill Road, Kenilworth,
New Jersey, USA 07033), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.

Re: running Hudi in AWS Glue Spark

["Mehrotra, Udit" <ud...@amazon.com.INVALID>]

Hi Jorge,

AWS Glue service itself does not support Hudi. However, you can use Glue as a metastore with Hudi on EMR. Hope that answers your question.

Thanks,
Udit Mehrotra
SDE | AWS EMR

On 3/6/20, 9:44 AM, "Vinoth Chandar" <vi...@apache.org> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
    
    
    
    https://aws.amazon.com/emr/features/hudi/ mentions that its integrated with
    the glue catalog.
    
    It should be similar to other datasources you use on Glue IIUC.. I have
    seen users talk about this on slack (IIRC)..
    Are you running into specific issues we can help with? May be the AWS folks
    here can chime in more?
    
    On Fri, Mar 6, 2020 at 3:47 AM Sanchez, Jorge
    <jo...@merck.com.invalid> wrote:
    
    > Hello,
    >
    > Did anybody tried to run Hudi within AWS Glue job, I searched the JIRA
    > issues but did not find anybody mentioning that.
    >
    >
    > Thanks,
    >
    > Jorge
    > Notice:  This e-mail message, together with any attachments, contains
    > information of Merck & Co., Inc. (2000 Galloping Hill Road, Kenilworth,
    > New Jersey, USA 07033), and/or its affiliates Direct contact information
    > for affiliates is available at
    > http://www.merck.com/contact/contacts.html) that may be confidential,
    > proprietary copyrighted and/or legally privileged. It is intended solely
    > for the use of the individual or entity named on this message. If you are
    > not the intended recipient, and have received this message in error,
    > please notify us immediately by reply e-mail and then delete it from
    > your system.
    >
    

Re: running Hudi in AWS Glue Spark

[Vinoth Chandar <vi...@apache.org>]

https://aws.amazon.com/emr/features/hudi/ mentions that its integrated with
the glue catalog.

It should be similar to other datasources you use on Glue IIUC.. I have
seen users talk about this on slack (IIRC)..
Are you running into specific issues we can help with? May be the AWS folks
here can chime in more?

On Fri, Mar 6, 2020 at 3:47 AM Sanchez, Jorge
<jo...@merck.com.invalid> wrote:

> Hello,
>
> Did anybody tried to run Hudi within AWS Glue job, I searched the JIRA
> issues but did not find anybody mentioning that.
>
>
> Thanks,
>
> Jorge
> Notice:  This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (2000 Galloping Hill Road, Kenilworth,
> New Jersey, USA 07033), and/or its affiliates Direct contact information
> for affiliates is available at
> http://www.merck.com/contact/contacts.html) that may be confidential,
> proprietary copyrighted and/or legally privileged. It is intended solely
> for the use of the individual or entity named on this message. If you are
> not the intended recipient, and have received this message in error,
> please notify us immediately by reply e-mail and then delete it from
> your system.
>