You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ellisms (via GitHub)" <gi...@apache.org> on 2025/04/20 21:58:07 UTC
[PR] Amazon EMR Sensors/Operators inherit AWS Base Classes [airflow]
ellisms opened a new pull request, #49486:
URL: https://github.com/apache/airflow/pull/49486
---
Updated EMR operators and Sensors to inherit the base AWS classes.
Partially addresses #35278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon EMR Sensors/Operators inherit AWS Base Classes [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #49486:
URL: https://github.com/apache/airflow/pull/49486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon EMR Sensors/Operators inherit AWS Base Classes [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #49486:
URL: https://github.com/apache/airflow/pull/49486#issuecomment-2822773894
This is now causing our example_emr_eks system test to fail:
```
[2025-04-22 21:45:13,663] {taskinstance.py:2457} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/airflow-core/src/airflow/models/taskinstance.py", line 2161, in _execute_callable
return create_executable_runner(
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/callback_runner.py", line 81, in run
return func(*args, **kwargs)
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 408, in wrapper
return func(self, *args, **kwargs)
File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/sensors/emr.py", line 319, in execute
super().execute(context=context)
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 408, in wrapper
return func(self, *args, **kwargs)
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/sensor.py", line 227, in execute
raise e
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/sensor.py", line 207, in execute
poke_return = self.poke(context)
File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/sensors/emr.py", line 304, in poke
state = self.hook.poll_query_status(
File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/hooks/emr.py", line 516, in poll_query_status
query_state = self.check_query_status(job_id)
File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/hooks/emr.py", line 488, in check_query_status
response = self.conn.describe_job_run(
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 980, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1047, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter virtualClusterId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon EMR Sensors/Operators inherit AWS Base Classes [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #49486:
URL: https://github.com/apache/airflow/pull/49486#issuecomment-2822781906
Testing a fix now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org