You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "aischeveva (via GitHub)" <gi...@apache.org> on 2023/07/06 12:22:28 UTC
[GitHub] [airflow] aischeveva opened a new issue, #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version > 7.1.0.
aischeveva opened a new issue, #32404:
URL: https://github.com/apache/airflow/issues/32404
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
### Airflow version
2.4.3, 2.5.0
### GlueJobOperator fails to create a job
With provider version 7.1.0 and above (I have also tried 7.4.1 and 8.2.0), GlueJobOperator doesn't recognize the type of the GlueVersion in create_job_kwargs correctly and fails to create a job. The passed parameter is string, but validation fails claiming that it's a float.
```
[2023-07-06, 10:52:27 UTC] {{taskinstance.py:1851}} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 146, in execute
glue_job_run = glue_job.initialize_job(self.script_args, self.run_job_kwargs)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 155, in initialize_job
job_name = self.create_or_update_glue_job()
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 303, in create_or_update_glue_job
self.update_job(**config)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 289, in update_job
self.get_conn().update_job(JobName=job_name, JobUpdate=job_kwargs)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/client.py", line 919, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/client.py", line 990, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/usr/local/airflow/.local/lib/python3.10/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter JobUpdate.GlueVersion, value: 3.0, type: <class 'float'>, valid types: <class 'str'>
```
I have observed this problem both in local setup with virtual environment on WSL Ubuntu 20.04 LTS with Airflow 2.5.0 and in Amazon MWAA with Airflow 2.4.3.
### What you think should happen instead
I would expect it shouldn't convert GlueVersion string to float when creating job to then fail its validation. The same code works fine with apache-airflow-providers-amazon==6.0.0, so I would expect same behavior.
### How to reproduce
Create a DAG and add a
```
GlueJobOperator(
task_id='submit_glue_job',
job_name='job-name',
script_location='script-location',
s3_bucket='your-s3-bucket',
iam_role_name='your-glue-role-name',
create_job_kwargs={"GlueVersion": "3.0"}
)
```
### Operating System
WSL Ubuntu 20.04 LTS / Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==7.1.0
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. [airflow]
Posted by "don1uppa (via GitHub)" <gi...@apache.org>.
don1uppa commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-2032657099
If it fix the value here back to a string under debug it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-1676664023
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. [airflow]
Posted by "don1uppa (via GitHub)" <gi...@apache.org>.
don1uppa commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-2032656200
well - its corrupted upfront. but I can't pin it down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. [airflow]
Posted by "don1uppa (via GitHub)" <gi...@apache.org>.
don1uppa commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-2031934173
Wow - I also had to go to version6 to get it to work - which sucks because I wanted the deferred operators feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-2031892830
> Still broken - I'll try the other versions as suggested
If it would still failing, please create [discussion](https://github.com/apache/airflow/discussions/new/choose) / [new issue](https://github.com/apache/airflow/issues/new?assignees=&labels=kind%3Abug%2Carea%3Aproviders%2Cneeds-triage&projects=&template=airflow_providers_bug_report.yml) with attached reproducible code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version > 7.1.0.
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-1623587413
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-1623984257
I'll also note that this particular use is tested in the system tests [here](https://github.com/apache/airflow/blob/main/tests/system/providers/amazon/aws/example_glue.py#L152) which [are passing](https://aws-mwaa.github.io/open-source/system-tests/dashboard.html), so I'm not sure if it's an Airflow issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. [airflow]
Posted by "don1uppa (via GitHub)" <gi...@apache.org>.
don1uppa commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-2031899634
I'll try to figure it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0. [airflow]
Posted by "don1uppa (via GitHub)" <gi...@apache.org>.
don1uppa commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-2031879378
Still broken - I'll try the other versions as suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
URL: https://github.com/apache/airflow/issues/32404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-1667015237
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aischeveva commented on issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
Posted by "aischeveva (via GitHub)" <gi...@apache.org>.
aischeveva commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-1623725411
> The stacktrace doesn't indicate issue with Glue operator. It suggest that type was changed in the upstream library.
I was thinking about it, but I tried running GlueJobOperator with providers 6.0.0 and then 7.1.0 while keeping botocore version the same ==1.29.56 in both cases, and with 6.0.0 it was still working as expected, but with 7.1.0 it failed. So I'm not quite sure what has changed between these versions.
I have also tried updating everything to the most recent versions, but I still get the same issue when the job doesn't exist yet (if the job already exists, it runs fine as it doesn't try to update it by default anymore).
```
boto3==1.27.1
botocore==1.30.1
apache-airflow-providers-amazon==8.2.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-1624862073
@aischeveva we are unable to reproduce the issue.
Please share a reproducable example from scratch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #32404: GlueJobOperator failing with Invalid type for parameter GlueVersion for provider version >= 7.1.0.
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #32404:
URL: https://github.com/apache/airflow/issues/32404#issuecomment-1623602105
The stacktrace doesn't indicate issue with Glue operator.
It suggest that type was changed in the upstream library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org