You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/06 13:59:47 UTC
[GitHub] [airflow] yoavfeld opened a new issue, #28159: Error when inherit from QuboleOperator
yoavfeld opened a new issue, #28159:
URL: https://github.com/apache/airflow/issues/28159
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
airflow version: 2.3
python version 3.7.13
problem: error occured when inheriting from QuboleOperator (airflow/providers/qubole/operators/qubole.py)
callstack:
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/qubole/operators/qubole.py", line 304, in __setattr__
if name in _get_template_fields(self):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/qubole/operators/qubole.py", line 312, in _get_template_fields
template_fields = object.__getattribute__(class_, 'template_fields')
AttributeError: 'BaseOperatorMeta' object has no attribute 'template_fields'
### What you think should happen instead
These changes in qubole.py file should be reverted:
https://github.com/apache/airflow/commit/33214d9326bb0bb52f06e230895f4f68fc952664#diff-63a6626367fb879a888263f6063e8aebb42d5a34e3a0f798fbb727b0a640aa56L270
https://github.com/apache/airflow/commit/33214d9326bb0bb52f06e230895f4f68fc952664#diff-63a6626367fb879a888263f6063e8aebb42d5a34e3a0f798fbb727b0a640aa56L279
### How to reproduce
Creating a class that inherit from QuboleOperator. for ex:
from airflow.providers.qubole.operators.qubole import QuboleOperator
class MyQuboleOperator(QuboleOperator):
def __init__(self):
super().__init__()
### Operating System
linux
### Versions of Apache Airflow Providers
airflow=2.3
apache-airflow-providers-qubole==3.3.1
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #28159: Error when inherit from QuboleOperator
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28159: Error when inherit from QuboleOperator
URL: https://github.com/apache/airflow/issues/28159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28159: Error when inherit from QuboleOperator
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28159:
URL: https://github.com/apache/airflow/issues/28159#issuecomment-1339468335
confirmed. cc @potiuk
I'm not sure why we even need the `__getattribute__` and `__setattr__` ?
https://github.com/apache/airflow/blob/78b8ea2f22239db3ef9976301234a66e50b47a94/airflow/providers/qubole/operators/qubole.py#L286-L299
it was added in https://github.com/apache/airflow/pull/1090/ I think we can just remove this part along with the `_get_template_fields`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28159: Error when inherit from QuboleOperator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28159:
URL: https://github.com/apache/airflow/issues/28159#issuecomment-1339432185
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28159: Error when inherit from QuboleOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28159:
URL: https://github.com/apache/airflow/issues/28159#issuecomment-1339633262
I don't think we should remove it. Looks like QuboleOpertor is precisely created in the way that template fields are SUPPOSED to be added when you derive from it.
This is what I understand from the comment in #1090 (and maybe @msumit still rememeber something that happened 6 years aago (!) and can confirm:
> I do agree with your concerns. Actually we didn't wanted to add dozens of new operators like QuboleHiveOperator, QuboleHadoopOperator etc, so we'd to do hacks like that. If we had individual command operators like I said above, then we could adhere to simple design like the other operators have.
I believe the initial idea was that The template_fields will be overridden by a "child" class (@yoavfeld what you should try, is you should likely define template_fields in your derived operator:
```
class MyQuboleOperator(QuboleOperator):
template_fields = ()
def init(self):
super().init()
````
I tihnk this should help you @yoavfeld and we should close the issue. Can you please confirm @yoavfeld that it solves your problem?
Converting into a discussion - unless we decide that this is an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org