You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/22 20:57:34 UTC
[GitHub] [airflow] wanderijames opened a new pull request #17178: Amazon EMR on Amazon EKS
wanderijames opened a new pull request #17178:
URL: https://github.com/apache/airflow/pull/17178
Adding operators for interacting with EMR on EKS as per https://github.com/apache/airflow/issues/17175
- Add operator for starting job
- Add operator for canceling a job
- Add operator for checking job state
closes: https://github.com/apache/airflow/issues/17175
related: https://github.com/apache/airflow/issues/17175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-885263138
We currently have one operator that allows us to run Spark job on Kubernetes. It works with both EKS and GCP as well as any other Kubernetes platform. - [SparkKubernetesOperator](https://github.com/apache/airflow/blob/d72b363929c86eb03fc9583002459bd10bc7eaeb/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L24). Why would anyone use this operator instead of the generic operator for Kubernetes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames commented on a change in pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames commented on a change in pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#discussion_r693494286
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_start_job.py
##########
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersStartJobOperator(BaseOperator):
+ """Operator to start a job.
+
+ A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query,
+ that you submit to Amazon EMR on EKS.
+
+ :param cluster_id: The ID of the virtual cluster for which the job run is submitted
+ :type cluster_id: str
+ :param execution_role_arn: The execution role ARN for the job run.
+ :type execution_role_arn: str
+ :param emr_release_label: The Amazon EMR release version to use for the job run.
+ :type emr_release_label: str
+ :param job_driver: The job driver for the job run.
+ :type job_driver: dict
+ :param configuration_overrides: The configuration overrides for the job run.
+ :type configuration_overrides: dict
+ :param tags: The tags assigned to job runs
+ :type tags: dict
+ :param name: The name of the job run.
+ :type name: str
+ :param client_token: The client idempotency token of the job run request. Provided if not populated
+ :type client_token: str
+ :param aws_conn_id: aws connection to uses
+ :type aws_conn_id: str
+ """
+
+ ui_color = '#f9c915'
+
+ def __init__(
+ self,
+ *,
+ cluster_id: str,
+ execution_role_arn: str,
+ emr_release_label: str,
+ job_driver: Dict[str, Any],
+ configuration_overrides: Optional[dict] = None,
+ tags: Optional[dict] = None,
+ name: Optional[str] = None,
+ client_token: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.start_job_params = dict(
+ cluster_id=cluster_id,
+ execution_role_arn=execution_role_arn,
+ emr_release_label=emr_release_label,
+ job_driver=job_driver,
+ configuration_overrides=configuration_overrides,
+ tags=tags,
+ name=name,
+ client_token=client_token,
Review comment:
Thanks @dacort for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames commented on a change in pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames commented on a change in pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#discussion_r682839880
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py
##########
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersGetJobStateOperator(BaseOperator):
Review comment:
Now that you asked me that question, I don't think this has a straight forward use case as an operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dacort commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
dacort commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-902837170
Here's my current thinking on this - I like the addition of the `Start` and `Cancel` operators. I think we should try to get #16766 merged and then you can 1. rebase off that commit and 2. add documentation to the `emr_eks.rst` file on how to use the two additional operators.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #17178:
URL: https://github.com/apache/airflow/pull/17178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dacort commented on a change in pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
dacort commented on a change in pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#discussion_r693067363
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_start_job.py
##########
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersStartJobOperator(BaseOperator):
+ """Operator to start a job.
+
+ A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query,
+ that you submit to Amazon EMR on EKS.
+
+ :param cluster_id: The ID of the virtual cluster for which the job run is submitted
+ :type cluster_id: str
+ :param execution_role_arn: The execution role ARN for the job run.
+ :type execution_role_arn: str
+ :param emr_release_label: The Amazon EMR release version to use for the job run.
+ :type emr_release_label: str
+ :param job_driver: The job driver for the job run.
+ :type job_driver: dict
+ :param configuration_overrides: The configuration overrides for the job run.
+ :type configuration_overrides: dict
+ :param tags: The tags assigned to job runs
+ :type tags: dict
+ :param name: The name of the job run.
+ :type name: str
+ :param client_token: The client idempotency token of the job run request. Provided if not populated
+ :type client_token: str
+ :param aws_conn_id: aws connection to uses
+ :type aws_conn_id: str
+ """
+
+ ui_color = '#f9c915'
+
+ def __init__(
+ self,
+ *,
+ cluster_id: str,
+ execution_role_arn: str,
+ emr_release_label: str,
+ job_driver: Dict[str, Any],
+ configuration_overrides: Optional[dict] = None,
+ tags: Optional[dict] = None,
+ name: Optional[str] = None,
+ client_token: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.start_job_params = dict(
+ cluster_id=cluster_id,
+ execution_role_arn=execution_role_arn,
+ emr_release_label=emr_release_label,
+ job_driver=job_driver,
+ configuration_overrides=configuration_overrides,
+ tags=tags,
+ name=name,
+ client_token=client_token,
Review comment:
Right, but it's not initialized anywhere. The default is `None`, which means if the client doesn't pass in a `client_token`, a new job won't get created after the first one. I ran into a similar issue and had to generate a UUID as the default:
```
client_token or str(uuid4())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-885423290
> We currently have one operator that allows us to run Spark job on Kubernetes. It works with both EKS and GCP as well as any other Kubernetes platform. - [SparkKubernetesOperator](https://github.com/apache/airflow/blob/d72b363929c86eb03fc9583002459bd10bc7eaeb/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L24). Why would anyone use this operator instead of the generic operator for Kubernetes?
Hey @mik-laj, I am aware of apache spark and livy operator and also EMR operator. However, EMR on EKS works differently because EMR launches virtual cluster in your EKS. The pods (spark master and executors) launched are ephemeral, only existing when a start job is invoked. For more information, kindly visit https://aws.amazon.com/emr/features/eks/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-937336230
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames edited a comment on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames edited a comment on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-885423290
> We currently have one operator that allows us to run Spark job on Kubernetes. It works with both EKS and GCP as well as any other Kubernetes platform. - [SparkKubernetesOperator](https://github.com/apache/airflow/blob/d72b363929c86eb03fc9583002459bd10bc7eaeb/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L24). Why would anyone use this operator instead of the generic operator for Kubernetes?
Hey @mik-laj, I am aware of apache spark and livy operator and also EMR operator. However, EMR on EKS works differently because EMR launches virtual cluster in your EKS. The pods (spark master and executors) launched are ephemeral, only existing when a start job is invoked. For more information, kindly visit https://aws.amazon.com/emr/features/eks/
In addition, SparkKubernetesOperator is only suitable if you have Spark cluster has been setup in Kubernetes. In this case, it is not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dacort commented on a change in pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
dacort commented on a change in pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#discussion_r675897221
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_start_job.py
##########
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersStartJobOperator(BaseOperator):
+ """Operator to start a job.
+
+ A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query,
+ that you submit to Amazon EMR on EKS.
+
+ :param cluster_id: The ID of the virtual cluster for which the job run is submitted
+ :type cluster_id: str
+ :param execution_role_arn: The execution role ARN for the job run.
+ :type execution_role_arn: str
+ :param emr_release_label: The Amazon EMR release version to use for the job run.
+ :type emr_release_label: str
+ :param job_driver: The job driver for the job run.
+ :type job_driver: dict
+ :param configuration_overrides: The configuration overrides for the job run.
+ :type configuration_overrides: dict
+ :param tags: The tags assigned to job runs
+ :type tags: dict
+ :param name: The name of the job run.
+ :type name: str
+ :param client_token: The client idempotency token of the job run request. Provided if not populated
+ :type client_token: str
+ :param aws_conn_id: aws connection to uses
+ :type aws_conn_id: str
+ """
+
+ ui_color = '#f9c915'
+
+ def __init__(
+ self,
+ *,
+ cluster_id: str,
+ execution_role_arn: str,
+ emr_release_label: str,
+ job_driver: Dict[str, Any],
+ configuration_overrides: Optional[dict] = None,
+ tags: Optional[dict] = None,
+ name: Optional[str] = None,
+ client_token: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.start_job_params = dict(
+ cluster_id=cluster_id,
+ execution_role_arn=execution_role_arn,
+ emr_release_label=emr_release_label,
+ job_driver=job_driver,
+ configuration_overrides=configuration_overrides,
+ tags=tags,
+ name=name,
+ client_token=client_token,
Review comment:
I don't see you defining `client_token` anywhere. Have you tested this in a live environment to validate that it works?
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py
##########
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersGetJobStateOperator(BaseOperator):
Review comment:
I'm curious what the use case is for an operator that just returns the job state?
Apologies if this is a silly question - still getting up to speed on typical Airflow patterns.
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py
##########
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersGetJobStateOperator(BaseOperator):
+ """Operator to get a job status.
+
+ A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query,
+ that you submit to Amazon EMR on EKS.
+
+ A job is in PENDING, SUBMITTED, RUNNING, FAILED, CANCELLED or CANCEL state
Review comment:
```suggestion
A job is in PENDING, SUBMITTED, RUNNING, FAILED, CANCELLED, CANCEL_PENDING, or COMPLETED state
```
Per https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_JobRun.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-903261091
> Here's my current thinking on this - I like the addition of the `Start` and `Cancel` operators. I think we should try to get #16766 merged and then you can 1. rebase off that commit and 2. add documentation to the `emr_eks.rst` file on how to use the two additional operators.
>
> Thoughts?
I agree @dacort , let's do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-937336230
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames commented on a change in pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames commented on a change in pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#discussion_r682838986
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_start_job.py
##########
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersStartJobOperator(BaseOperator):
+ """Operator to start a job.
+
+ A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query,
+ that you submit to Amazon EMR on EKS.
+
+ :param cluster_id: The ID of the virtual cluster for which the job run is submitted
+ :type cluster_id: str
+ :param execution_role_arn: The execution role ARN for the job run.
+ :type execution_role_arn: str
+ :param emr_release_label: The Amazon EMR release version to use for the job run.
+ :type emr_release_label: str
+ :param job_driver: The job driver for the job run.
+ :type job_driver: dict
+ :param configuration_overrides: The configuration overrides for the job run.
+ :type configuration_overrides: dict
+ :param tags: The tags assigned to job runs
+ :type tags: dict
+ :param name: The name of the job run.
+ :type name: str
+ :param client_token: The client idempotency token of the job run request. Provided if not populated
+ :type client_token: str
+ :param aws_conn_id: aws connection to uses
+ :type aws_conn_id: str
+ """
+
+ ui_color = '#f9c915'
+
+ def __init__(
+ self,
+ *,
+ cluster_id: str,
+ execution_role_arn: str,
+ emr_release_label: str,
+ job_driver: Dict[str, Any],
+ configuration_overrides: Optional[dict] = None,
+ tags: Optional[dict] = None,
+ name: Optional[str] = None,
+ client_token: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.start_job_params = dict(
+ cluster_id=cluster_id,
+ execution_role_arn=execution_role_arn,
+ emr_release_label=emr_release_label,
+ job_driver=job_driver,
+ configuration_overrides=configuration_overrides,
+ tags=tags,
+ name=name,
+ client_token=client_token,
Review comment:
It is being used in line 89;
```python
response = emr_containers.start_job(**self.start_job_params)
```
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py
##########
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersGetJobStateOperator(BaseOperator):
Review comment:
Now that you asked me that question, I don't think this has a straigh forward use case as an operator.
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py
##########
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersGetJobStateOperator(BaseOperator):
Review comment:
Now that you asked me that question, I don't think this has a straight forward use case as an operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-885227822
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
eladkal commented on pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#issuecomment-962417208
@wanderijames https://github.com/apache/airflow/pull/16766 was merged. Would you like to proceed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames commented on a change in pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames commented on a change in pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#discussion_r682839880
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py
##########
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersGetJobStateOperator(BaseOperator):
Review comment:
Now that you asked me that question, I don't think this has a straigh forward use case as an operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wanderijames commented on a change in pull request #17178: Amazon EMR on Amazon EKS
Posted by GitBox <gi...@apache.org>.
wanderijames commented on a change in pull request #17178:
URL: https://github.com/apache/airflow/pull/17178#discussion_r682838986
##########
File path: airflow/providers/amazon/aws/operators/emr_containers_start_job.py
##########
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersStartJobOperator(BaseOperator):
+ """Operator to start a job.
+
+ A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query,
+ that you submit to Amazon EMR on EKS.
+
+ :param cluster_id: The ID of the virtual cluster for which the job run is submitted
+ :type cluster_id: str
+ :param execution_role_arn: The execution role ARN for the job run.
+ :type execution_role_arn: str
+ :param emr_release_label: The Amazon EMR release version to use for the job run.
+ :type emr_release_label: str
+ :param job_driver: The job driver for the job run.
+ :type job_driver: dict
+ :param configuration_overrides: The configuration overrides for the job run.
+ :type configuration_overrides: dict
+ :param tags: The tags assigned to job runs
+ :type tags: dict
+ :param name: The name of the job run.
+ :type name: str
+ :param client_token: The client idempotency token of the job run request. Provided if not populated
+ :type client_token: str
+ :param aws_conn_id: aws connection to uses
+ :type aws_conn_id: str
+ """
+
+ ui_color = '#f9c915'
+
+ def __init__(
+ self,
+ *,
+ cluster_id: str,
+ execution_role_arn: str,
+ emr_release_label: str,
+ job_driver: Dict[str, Any],
+ configuration_overrides: Optional[dict] = None,
+ tags: Optional[dict] = None,
+ name: Optional[str] = None,
+ client_token: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.start_job_params = dict(
+ cluster_id=cluster_id,
+ execution_role_arn=execution_role_arn,
+ emr_release_label=emr_release_label,
+ job_driver=job_driver,
+ configuration_overrides=configuration_overrides,
+ tags=tags,
+ name=name,
+ client_token=client_token,
Review comment:
It is being used in line 89;
```python
response = emr_containers.start_job(**self.start_job_params)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org