You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/11 01:40:34 UTC

[GitHub] [airflow] gumartinm opened a new issue #20215: EMR serverless, new operator

gumartinm opened a new issue #20215:
URL: https://github.com/apache/airflow/issues/20215


   ### Description
   
   A new EMR serverless has been announced and it is already available, see:
   
   - https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/
   - https://aws.amazon.com/emr/serverless/
   
   Having an operator for creating applications and submitting jobs to EMR serverless would be awesome.
   
   ### Use case/motivation
   
   New operator for working with EMR serverless.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ferruzzi commented on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-996172343


   That's correct, this is in preview and there is no announced date for general release yet.  I'll also be happy to help once the docs are available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dacort commented on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
dacort commented on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-1081184838


   I just published some information about how to use (a custom, while in preview) boto3 to access the EMR Serverless API.
   
   https://github.com/aws-samples/emr-serverless-samples/tree/main/examples/python-api


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] yeshbash commented on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
yeshbash commented on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-992116575


   Looks like the feature is not Generally Available and I couldn't find the docs to the APIs. I'd be happy to pitch in with this operator if there are more details about it. Also, any guidance from the community is appreciated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-991406055


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] TreyYi edited a comment on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
TreyYi edited a comment on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-1060194159


   Hi @dacort,
   
   I am also interested in EMR Serverless features for airflow such as operators, sensors, or testings.
   Once EMR serverless is released officially, I will try to provide some suggestions, and I am willing to help you even if it wouldn't be my PR!
   (FYI, currently I am in data engineer position for operating EMR clusters and airflow server.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-991406055


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] TreyYi commented on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
TreyYi commented on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-1060194159


   Hi @dacort,
   
   I am also interested in EMR Serverless features for airflow such as operators, sensors, or testings.
   Once EMR serverless is released officially, I will try to provide some suggestions, and I am willing to help you even if it wouldn't be my PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dacort commented on issue #20215: EMR serverless, new operator

Posted by GitBox <gi...@apache.org>.
dacort commented on issue #20215:
URL: https://github.com/apache/airflow/issues/20215#issuecomment-1013392808


   @yeshbash @ferruzzi If you haven't already, please sign up for the preview here: https://pages.awscloud.com/EMR-Serverless-Preview.html
   
   We've started to let folks in and you will receive info on how to access the preview when you're allowlisted.
   
   Docs are [here](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html) and I've also created an [emr-serverless-samples](https://github.com/aws-samples/emr-serverless-samples) repository that may be useful.
   
   I'm on the EMR team and would be happy to help implement or review any PRs. 😁


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org