You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Lee-W (via GitHub)" <gi...@apache.org> on 2023/05/23 14:53:02 UTC

[GitHub] [airflow] Lee-W opened a new issue, #31480: Missing LogUri from emr describe-cluster API when executing EmrCreateJobFlowOperator

Lee-W opened a new issue, #31480:
URL: https://github.com/apache/airflow/issues/31480

   ### Apache Airflow version
   
   main (development)
   
   ### What happened
   
   Encounter the following error when executing `EmrCreateJobFlowOperator`
   
   ```
   Traceback (most recent call last):
     File "/usr/local/lib/python3.10/site-packages/airflow/providers/amazon/aws/operators/emr.py", line 695, in execute
       log_uri=get_log_uri(emr_client=self._emr_hook.conn, job_flow_id=self._job_flow_id),
     File "/usr/local/lib/python3.10/site-packages/airflow/providers/amazon/aws/links/emr.py", line 61, in get_log_uri
       log_uri = S3Hook.parse_s3_url(response["Cluster"]["LogUri"])
   KeyError: 'LogUri'
   ```
   
   According to the [this document](https://docs.aws.amazon.com/cli/latest/reference/emr/describe-cluster.html), it seems we might not always be able to get `["Cluster"]["LogUri"]`  and we encounter errors after the release in https://github.com/apache/airflow/issues/31322.
   
   ### What you think should happen instead
   
   The `EmrCreateJobFlowOperator` should finish execution without error.
   
   ### How to reproduce
   
   1. `git clone github.com/apache/airflow/`
   2. `cd airflow`
   3. `git checkout c082aec089405ed0399cfee548011b0520be0011` (the main branch when I found this issue)
   4. Add the following DAG to `files/dags/` and name it as `example_emr.py`
   
   
   ```python
   import os
   from datetime import datetime, timedelta
   
   from airflow import DAG
   from airflow.providers.amazon.aws.operators.emr import EmrCreateJobFlowOperator, EmrTerminateJobFlowOperator
   
   
   JOB_FLOW_OVERRIDES = {
       "Name": "example_emr_sensor_cluster",
       "ReleaseLabel": "emr-5.29.0",
       "Applications": [{"Name": "Spark"}],
       "Instances": {
           "InstanceGroups": [
               {
                   "Name": "Primary node",
                   "Market": "ON_DEMAND",
                   "InstanceRole": "MASTER",
                   "InstanceType": "m4.large",
                   "InstanceCount": 1,
               },
           ],
           "KeepJobFlowAliveWhenNoSteps": False,
           "TerminationProtected": False,
       },
       "JobFlowRole": "EMR_EC2_DefaultRole"
       "ServiceRole": "EMR_DefaultRole",
   }
   
   DEFAULT_ARGS = {
       "execution_timeout": timedelta(hours=6),
       "retries": 2,
       "retry_delay": 60,
   }
   
   
   with DAG(
       dag_id="example_emr_sensor",
       schedule=None,
       start_date=datetime(2022, 1, 1),
       default_args=DEFAULT_ARGS,
       catchup=False,
   ) as dag:
       create_job_flow = EmrCreateJobFlowOperator(
           task_id="create_job_flow",
           job_flow_overrides=JOB_FLOW_OVERRIDES,
           aws_conn_id="aws_default",
       )
   
       remove_job_flow = EmrTerminateJobFlowOperator(
           task_id="remove_job_flow",
           job_flow_id=create_job_flow.output,
           aws_conn_id="aws_default",
           trigger_rule="all_done",
       )
   
       create_job_flow >> remove_job_flow
   ```
   
   5. `breeze --python 3.8 --backend sqlite start-airflow`
   6. Trigger the DAG from web UI
   
   ### Operating System
   
   mac OS 13.4
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   https://github.com/apache/airflow/issues/31322#issuecomment-1556876252
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] hussein-awala commented on issue #31480: Missing LogUri from emr describe-cluster API when executing EmrCreateJobFlowOperator

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #31480:
URL: https://github.com/apache/airflow/issues/31480#issuecomment-1560723485

   > According to the [this document](https://docs.aws.amazon.com/cli/latest/reference/emr/describe-cluster.html), it seems we might not always be able to get ["Cluster"]["LogUri"]
   
   The `LogUri` is returned only when logging option is enabled in the cluster, but the users can keep it disabled, in this case they will get an exception as yours


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #31480: Missing LogUri from emr describe-cluster API when executing EmrCreateJobFlowOperator

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #31480:
URL: https://github.com/apache/airflow/issues/31480#issuecomment-1559596992

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] hussein-awala closed issue #31480: Missing LogUri from emr describe-cluster API when executing EmrCreateJobFlowOperator

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed issue #31480: Missing LogUri from emr describe-cluster API when executing EmrCreateJobFlowOperator
URL: https://github.com/apache/airflow/issues/31480


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org