You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/02 18:46:40 UTC
[GitHub] [airflow] andormarkus opened a new issue #19372: EKSCreateNodegroupOperator - extra kwargs
andormarkus opened a new issue #19372:
URL: https://github.com/apache/airflow/issues/19372
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==2.3.0
### Apache Airflow version
2.1.4 (latest released)
### Operating System
Amazon Linux 2
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
Adding extra kwargs to EKSCreateNodegroupOperator brokes the DAG.
```shell
Broken DAG: [/opt/airflow/dags/maintenance_deltatable_compaction.py] Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 188, in apply_defaults
result = func(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 535, in __init__
raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to EKSCreateNodegroupOperator (task_id: create_eks_nodegroup). Invalid arguments were:
**kwargs: {'amiType': 'BOTTLEROCKET_x86_64', 'instanceTypes': 't3.large', 'capacityType': 'ON_DEMAND'}
```
### What you expected to happen
I should be able to pass all the kwargs which are available in boto3 into EKS operator.
### How to reproduce
```python
create_nodegroup = EKSCreateNodegroupOperator(
task_id='create_eks_nodegroup',
cluster_name=cluster_name,
nodegroup_name=node_group_name,
nodegroup_subnets=get_subnets(),
nodegroup_role_arn=get_node_role(),
amiType = "BOTTLEROCKET_x86_64",
instanceTypes = "t3.large",
capacityType = "ON_DEMAND"
)
````
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-959556784
Thanks, we discussed it a bit in Slack. I have two PRs in the works at the moment, I will have a look at this one next.
It's an interesting problem. At a glance, it looks like the base pod operator will throw a fit if it gets kwargs it doesn't expect so I/we will need to sanitize the kwargs before passing them there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19372:
URL: https://github.com/apache/airflow/issues/19372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-970767266
Actually, it looks like this is sorted. As of Airflow 2.0 there is an operator parameter to allow illegal arguments: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#allow-illegal-arguments
@andormarkus - my test environment is currently broken so I can't test it, but can you try adding `allow_illegal_argument = True` (or perhaps ALLOW_ILLEGAL_ARGUMENTS??) to the operator and see if that does it for you?
```
create_nodegroup = EKSCreateNodegroupOperator(
task_id='create_eks_nodegroup',
cluster_name=cluster_name,
nodegroup_name=node_group_name,
nodegroup_subnets=get_subnets(),
nodegroup_role_arn=get_node_role(),
amiType = "BOTTLEROCKET_x86_64",
instanceTypes = "t3.large",
capacityType = "ON_DEMAND",
allow_illegal_argument = True
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-959579938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971319085
Hi @ferruzzi
None of them worked.
`allow_illegal_argument = True`
```shell
Broken DAG: [/opt/airflow/dags/eks.py] Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 188, in apply_defaults
result = func(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 535, in __init__
raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to EKSCreateNodegroupOperator (task_id: create_eks_nodegroup). Invalid arguments were:
**kwargs: {'amiType': 'BOTTLEROCKET_x86_64', 'instanceTypes': 't3.large', 'capacityType': 'ON_DEMAND', 'allow_illegal_argument': True}
```
`AIRFLOW__OPERATORS__ALLOW_ILLEGAL_ARGUMENTS: 'true'`
I have requested `t3.large` got `t3.medium` and the AMI is off as well
<img width="1601" alt="Screen Shot 2021-11-17 at 08 41 43" src="https://user-images.githubusercontent.com/51825189/142156989-4b87756c-d7c1-45a7-ad9d-458427c1f0e7.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk removed a comment on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
potiuk removed a comment on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971437006
What does your `airflow config list` say when you run it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971426098
Hi @potiuk
In the official docker-compose file I set the following env variable.
```yaml
environment:
AIRFLOW__OPERATORS__ALLOW_ILLEGAL_ARGUMENTS: True
```
I got not invalid arguments AirflowException but the operator bring up incorrectly configured nodegroup (see my previous comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-958060215
CC: @ferruzzi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971438088
Ah. I see. Then I think this is the problem with the parameters you are passing to the operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971437006
What does your `airflow config list` say when you run it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-959556784
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-959579938
As a solution I can try to submit a PR when I declare all the kwargs from boto3, however I don't know does it matches with the design of the EKS operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971376702
Don't set `allow_illegal_argumet" as parameter. just override it in airflow config.
It should work to set `AIRFLOW__OPERATORS__ALLOW_ILLEGAL_ARGUMENTS` should be set to `True` or update the `airflow.cfg' to avoid the exception.
You can verify if it is properly set by exec into the container and running `airflow config list`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971445094
Unfortunately, I do not know too much about this operator, but passing parameters this way seems completely wrong.
When you look at the code of the operator https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/eks.py#L278
It just uses those parameters - and you are trying to pass some new ones.
```
eks_hook.create_nodegroup(
clusterName=self.cluster_name,
nodegroupName=self.nodegroup_name,
subnets=self.nodegroup_subnets,
nodeRole=self.nodegroup_role_arn,
)
```
Again - I do not know how to pass those parameters ( I do not know EKS, but either those should be passed somehow using the existing parameters in this call, or the operator needs to be extended to support more parameters.
Closing this as invalid as this is not a bug, ideally if you find a way how to pass it - maybe you could improve the docs of the operator, or if some changes are needed, a PR with adding more parameters is most welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-971802684
@potiuk - thanks for the correction, I still have a lot to learn :)
@andormarkus - I see, I see. So these are kwargs that need to be added to the hook, then to the operator. I get it now. Sorry for the confusion. (( Moving to the new thread after this reply. ))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-959556784
Thanks, we discussed it a bit in Slack. I have two PRs in the works at the moment, I will have a look at this one next.
It's an interesting problem. At a glance, it looks like the base pod operator will throw a fit if it gets kwargs it doesn't expect so I/we will need to sanitize the kwargs before passing them there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-958060215
CC: @ferruzzi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #19372: EKSCreateNodegroupOperator - extra kwargs
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #19372:
URL: https://github.com/apache/airflow/issues/19372#issuecomment-959579938
As a solution I can try to submit a PR when I declare all the kwargs from boto3, however I don't know does it matches with the design of the EKS operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org