You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/27 07:10:35 UTC
[GitHub] [airflow] chandansingh1653 opened a new issue #9543: Enable airflow to orchestrate python job on AWS glue
chandansingh1653 opened a new issue #9543:
URL: https://github.com/apache/airflow/issues/9543
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
Description
Enable airflow to trigger the python glue job
-->
Use case:
<!-- Glue jobs are used to schedule the python and spark job, the current Glue job operator only runs the spark job.
There should be an option given to user which type of code they want to orchestrate using glue job operator i.e Pyspark or Python.
Documentation for glue: https://docs.aws.amazon.com/glue/latest/dg/add-job.html
Capabilities this will add: Airflow would be able to orchestrate the long-running python job using Glue
-->
**Related Issues**
<!-- Is there currently another issue associated with this? -->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chandansingh1653 commented on issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
chandansingh1653 commented on issue #9543:
URL: https://github.com/apache/airflow/issues/9543#issuecomment-650512547
I can work on this if this feature looks valid
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9543:
URL: https://github.com/apache/airflow/issues/9543#issuecomment-650711380
https://github.com/apache/airflow/blob/c420dbd6e13e17867eb4ccc4271b37966310ac0f/airflow/providers/amazon/aws/operators/glue.py#L32
I think it is supported. Did I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9543:
URL: https://github.com/apache/airflow/issues/9543#issuecomment-650511973
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #9543:
URL: https://github.com/apache/airflow/issues/9543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9543:
URL: https://github.com/apache/airflow/issues/9543#issuecomment-650736276
I am not an AWS expert, but I will trust you that the situation looks as you say. I assigned you to this ticket and I look forward to your contribution. If we are wrong, then the reviewer of this change will discuss this topic again.
> If you create pull-request, you don't have to create an issue first, but if you want, you can do it. Creating an issue will allow you to collect feedback or share plans with other people.
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chandansingh1653 commented on issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
chandansingh1653 commented on issue #9543:
URL: https://github.com/apache/airflow/issues/9543#issuecomment-650735368
Current AWS Hook doesn't run python shell .AWS glue job operator by default will run spark jobs.However if user wants to run python shell on AWS glue its not possible using current AWS glue job operator
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chandansingh1653 edited a comment on issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
chandansingh1653 edited a comment on issue #9543:
URL: https://github.com/apache/airflow/issues/9543#issuecomment-650735368
Current AWS Hook doesn't run python shell .AWS glue job operator by default will run spark jobs.However if user wants to run python shell on AWS glue its not possible using current AWS glue job operator.
https://github.com/apache/airflow/blob/c420dbd6e13e17867eb4ccc4271b37966310ac0f/airflow/providers/amazon/aws/hooks/glue.py#L171
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #9543: Enable airflow to orchestrate python job on AWS glue
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #9543:
URL: https://github.com/apache/airflow/issues/9543#issuecomment-962418035
Since the original author of the request did not pursue with it and the needed modification is not very clear. I'm not confident that someone else can pick up this task plus this issue had no activity for more than a year.
I'll close it for now. If someone want to work on it you can directly open a PR or comment and explain further the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org