You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sid <fl...@gmail.com> on 2022/06/23 05:35:25 UTC

Need help with the configuration for AWS glue jobs

Hi Team,

Could anyone help me in the below problem:

https://stackoverflow.com/questions/72724999/how-to-calculate-number-of-g-1-workers-in-aws-glue-for-processing-1tb-data

Thanks,
Sid

Re: Need help with the configuration for AWS glue jobs

Posted by Sid <fl...@gmail.com>.

Where can I find information on the size of the datasets supported by AWS
Glue? I didn't see it on the documentation

Also, if I want to process TBs of data for eg 1TB what should be the ideal
EMR cluster configuration?

Could you please guide me on this?

Thanks,
Sid.

On Thu, 23 Jun 2022, 23:44 Gourav Sengupta, <go...@gmail.com>
wrote:

> Please use EMR, Glue is not made for heavy processing jobs.
>
> On Thu, Jun 23, 2022 at 6:36 AM Sid <fl...@gmail.com> wrote:
>
>> Hi Team,
>>
>> Could anyone help me in the below problem:
>>
>>
>> https://stackoverflow.com/questions/72724999/how-to-calculate-number-of-g-1-workers-in-aws-glue-for-processing-1tb-data
>>
>> Thanks,
>> Sid
>>
>

Re: Need help with the configuration for AWS glue jobs

Posted by Gourav Sengupta <go...@gmail.com>.

Please use EMR, Glue is not made for heavy processing jobs.

On Thu, Jun 23, 2022 at 6:36 AM Sid <fl...@gmail.com> wrote:

> Hi Team,
>
> Could anyone help me in the below problem:
>
>
> https://stackoverflow.com/questions/72724999/how-to-calculate-number-of-g-1-workers-in-aws-glue-for-processing-1tb-data
>
> Thanks,
> Sid
>