You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Craig Rodrigues <cr...@gmail.com> on 2018/05/30 00:11:52 UTC
KubernetesPodOperator: Invalid arguments were passed to BaseOperator
I tested master branch by putting the following in my requirements.txt:
git+https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
and did a pip install -r requirements.txt
When I started the airflow webserver, I saw deprecation warnings. I
put some additional debugging in models.py to through an exception so that I could see the
full stacktrace:
[2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import: /Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/example_dags/example_kubernetes_operator.py
Traceback (most recent call last):
File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py", line 304, in process_file
m = imp.load_source(mod_name, filepath)
File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/example_dags/example_kubernetes_operator.py", line 53, in <module>
dag=dag)
File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/utils/decorators.py", line 98, in wrapper
result = func(*args, **kwargs)
File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py", line 2308, in __init__
raise Exception("Invalid use of args or kwargs")
Exception: Invalid use of args or kwargs
If looks like example_kubernetes_operator.py, this code is the source of the exception:
k = KubernetesPodOperator(
namespace='default',
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["echo", "10"],
labels={"foo": "bar"},
name="airflow-test-pod",
in_cluster=False,
task_id="task",
get_logs=True,
dag=dag)
Without my extra debugging, the deprecation warning looks like this:
[2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING - Could not import KubernetesPodOperator
/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py:2315: PendingDeprecationWarning: Invalid arguments were passed to BaseOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were:
*args: ()
**kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04', 'labels': {'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'], 'arguments': ['echo', '10'], 'in_cluster': False, 'get_logs': True}
category=PendingDeprecationWarning
What is the correct fix for this? It looks like a lot of operators pass in arguments which are not
processed by BaseOperator, and thus trip over this deprecation warning.
--
Craig
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by Craig Rodrigues <cr...@gmail.com>.
For this particular DeprecationWarning, this problem is not caused
by dependencies on kubernetes stuff.
On this line:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/kubernetes_pod_operator.py#L137
The __init__() method for KubernetesPodOperator is calling the __init__()
method
for BaseOperator, and passing in values for kwargs that BaseOperator
doesn't accept:
name': 'airflow-test-pod',
'image': 'ubuntu:16.04',
'labels': {'foo': 'bar'},
'namespace': 'default',
'cmds': ['bash', '-cx'],
'arguments': ['echo', '10'],
'in_cluster': False,
'get_logs': True
I don't understand how these things are being passed via kwargs?
--
Craig
On Wed, May 30, 2018 at 1:40 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Craig,
>
> This is something that needs to be fixed. I agree with you this is very
> dirty. In your installation you're not installing the kubernetes stuff, so
> the KubernetesPodOperator is ignored. We need to figure out how to have
> example dags that are not compatible with the vanilla installation, or we
> need to remove the kubernetes example for now, and move it to the
> documentation.
>
> Cheers, Fokko
>
--
Craig Rodrigues
rodrigc@rodrigues.org
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Taylor,
Thanks, I was thinking about something similar like you're suggesting. But
I'm not confident if the sys.exit() won't kill the whole Airflow process.
For example, if you do an airflow initdb, also the examples will be
initialised, and if you don't have kubernetes installed, it will hit the
sys.exit().
Cheers, Fokko
2018-05-30 20:08 GMT+02:00 Taylor Edmiston <te...@gmail.com>:
> I used requests instead of kube as an example, but what do you think about
> doing something like this? I'm happy to put this into a PR if it would
> solve the pain point today.
>
> import logging
>
> try:
> import requests
> except ModuleNotFoundError:
> import sys
> logging.warning('kube not installed - skipping kube examples')
> sys.exit()
>
> resp = requests.get('http://example.com')
> print(resp)
> ...
>
> Taylor
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/> | CV
> <https://stackoverflow.com/cv/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston>
>
>
> On Wed, May 30, 2018 at 4:40 AM, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Hi Craig,
> >
> > This is something that needs to be fixed. I agree with you this is very
> > dirty. In your installation you're not installing the kubernetes stuff,
> so
> > the KubernetesPodOperator is ignored. We need to figure out how to have
> > example dags that are not compatible with the vanilla installation, or we
> > need to remove the kubernetes example for now, and move it to the
> > documentation.
> >
> > Cheers, Fokko
> >
> > 2018-05-30 2:11 GMT+02:00 Craig Rodrigues <cr...@gmail.com>:
> >
> > > I tested master branch by putting the following in my requirements.txt:
> > >
> > > git+https://github.com/rodrigc/incubator-airflow@
> > > master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,
> > > ldap,mysql,postgres,redis,slack,s3]
> > >
> > > and did a pip install -r requirements.txt
> > >
> > > When I started the airflow webserver, I saw deprecation warnings. I
> > > put some additional debugging in models.py to through an exception so
> > that
> > > I could see the
> > > full stacktrace:
> > >
> > > [2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import:
> > > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > > airflow/example_dags/example_kubernetes_operator.py
> > > Traceback (most recent call last):
> > > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/models.py",
> > > line 304, in process_file
> > > m = imp.load_source(mod_name, filepath)
> > > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > > airflow/example_dags/example_kubernetes_operator.py", line 53, in
> > <module>
> > > dag=dag)
> > > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/utils/decorators.py",
> > > line 98, in wrapper
> > > result = func(*args, **kwargs)
> > > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/models.py",
> > > line 2308, in __init__
> > > raise Exception("Invalid use of args or kwargs")
> > > Exception: Invalid use of args or kwargs
> > >
> > >
> > > If looks like example_kubernetes_operator.py, this code is the source
> of
> > > the exception:
> > >
> > > k = KubernetesPodOperator(
> > > namespace='default',
> > > image="ubuntu:16.04",
> > > cmds=["bash", "-cx"],
> > > arguments=["echo", "10"],
> > > labels={"foo": "bar"},
> > > name="airflow-test-pod",
> > > in_cluster=False,
> > > task_id="task",
> > > get_logs=True,
> > > dag=dag)
> > >
> > >
> > > Without my extra debugging, the deprecation warning looks like this:
> > >
> > > [2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING
> -
> > > Could not import KubernetesPodOperator
> > > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/models.py:2315:
> > > PendingDeprecationWarning: Invalid arguments were passed to
> BaseOperator.
> > > Support for passing such arguments will be dropped in Airflow 2.0.
> > Invalid
> > > arguments were:
> > > *args: ()
> > > **kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04',
> 'labels':
> > > {'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'],
> > > 'arguments': ['echo', '10'], 'in_cluster': False, 'get_logs': True}
> > > category=PendingDeprecationWarning
> > >
> > >
> > >
> > > What is the correct fix for this? It looks like a lot of operators
> pass
> > > in arguments which are not
> > > processed by BaseOperator, and thus trip over this deprecation warning.
> > >
> > > --
> > > Craig
> > >
> >
>
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by Taylor Edmiston <te...@gmail.com>.
I used requests instead of kube as an example, but what do you think about
doing something like this? I'm happy to put this into a PR if it would
solve the pain point today.
import logging
try:
import requests
except ModuleNotFoundError:
import sys
logging.warning('kube not installed - skipping kube examples')
sys.exit()
resp = requests.get('http://example.com')
print(resp)
...
Taylor
*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>
On Wed, May 30, 2018 at 4:40 AM, Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Craig,
>
> This is something that needs to be fixed. I agree with you this is very
> dirty. In your installation you're not installing the kubernetes stuff, so
> the KubernetesPodOperator is ignored. We need to figure out how to have
> example dags that are not compatible with the vanilla installation, or we
> need to remove the kubernetes example for now, and move it to the
> documentation.
>
> Cheers, Fokko
>
> 2018-05-30 2:11 GMT+02:00 Craig Rodrigues <cr...@gmail.com>:
>
> > I tested master branch by putting the following in my requirements.txt:
> >
> > git+https://github.com/rodrigc/incubator-airflow@
> > master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,
> > ldap,mysql,postgres,redis,slack,s3]
> >
> > and did a pip install -r requirements.txt
> >
> > When I started the airflow webserver, I saw deprecation warnings. I
> > put some additional debugging in models.py to through an exception so
> that
> > I could see the
> > full stacktrace:
> >
> > [2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import:
> > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/example_dags/example_kubernetes_operator.py
> > Traceback (most recent call last):
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/models.py",
> > line 304, in process_file
> > m = imp.load_source(mod_name, filepath)
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/example_dags/example_kubernetes_operator.py", line 53, in
> <module>
> > dag=dag)
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/utils/decorators.py",
> > line 98, in wrapper
> > result = func(*args, **kwargs)
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/models.py",
> > line 2308, in __init__
> > raise Exception("Invalid use of args or kwargs")
> > Exception: Invalid use of args or kwargs
> >
> >
> > If looks like example_kubernetes_operator.py, this code is the source of
> > the exception:
> >
> > k = KubernetesPodOperator(
> > namespace='default',
> > image="ubuntu:16.04",
> > cmds=["bash", "-cx"],
> > arguments=["echo", "10"],
> > labels={"foo": "bar"},
> > name="airflow-test-pod",
> > in_cluster=False,
> > task_id="task",
> > get_logs=True,
> > dag=dag)
> >
> >
> > Without my extra debugging, the deprecation warning looks like this:
> >
> > [2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING -
> > Could not import KubernetesPodOperator
> > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/models.py:2315:
> > PendingDeprecationWarning: Invalid arguments were passed to BaseOperator.
> > Support for passing such arguments will be dropped in Airflow 2.0.
> Invalid
> > arguments were:
> > *args: ()
> > **kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04', 'labels':
> > {'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'],
> > 'arguments': ['echo', '10'], 'in_cluster': False, 'get_logs': True}
> > category=PendingDeprecationWarning
> >
> >
> >
> > What is the correct fix for this? It looks like a lot of operators pass
> > in arguments which are not
> > processed by BaseOperator, and thus trip over this deprecation warning.
> >
> > --
> > Craig
> >
>
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by Craig Rodrigues <cr...@gmail.com>.
I have submitted a patch:
https://github.com/apache/incubator-airflow/pull/3442
--
Craig
On 2018/05/30 19:45:23, Craig Rodrigues <cr...@gmail.com> wrote:
> Oh, OK, I just saw this in example_kubernetes_operator.py:
>
> try:
> # Kubernetes is optional, so not available in vanilla Airflow
> # pip install airflow[gcp]
> from airflow.contrib.operators.kubernetes_pod_operator import
> KubernetesPodOperator
> except ImportError:
> # Just import the BaseOperator as the KubernetesPodOperator
> logging.warn("Could not import KubernetesPodOperator")
> from airflow.models import BaseOperator as KubernetesPodOperator
>
> That code is just bad and confusing for an example!!
>
> --
> Craig
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by Craig Rodrigues <cr...@gmail.com>.
Oh, OK, I just saw this in example_kubernetes_operator.py:
try:
# Kubernetes is optional, so not available in vanilla Airflow
# pip install airflow[gcp]
from airflow.contrib.operators.kubernetes_pod_operator import
KubernetesPodOperator
except ImportError:
# Just import the BaseOperator as the KubernetesPodOperator
logging.warn("Could not import KubernetesPodOperator")
from airflow.models import BaseOperator as KubernetesPodOperator
That code is just bad and confusing for an example!!
--
Craig
On Wed, May 30, 2018 at 12:39 PM Chris Palmer <ch...@crpalmer.com> wrote:
> In the example
> <
> https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_kubernetes_operator.py#L24
> >
> it imports BaseOperator as KubernetesPodOperator, when the kubernetes
> modules can't be found.
>
> On Wed, May 30, 2018 at 3:34 PM, Craig Rodrigues <ro...@crodrigues.org>
> wrote:
>
> > For this particular DeprecationWarning, this problem is not caused
> > by dependencies on kubernetes stuff.
> >
> > On this line:
> > https://github.com/apache/incubator-airflow/blob/master/
> > airflow/contrib/operators/kubernetes_pod_operator.py#L137
> >
> > The __init__() method for KubernetesPodOperator is calling the __init__()
> > method
> > for BaseOperator, and passing in values for kwargs that BaseOperator
> > doesn't accept:
> >
> > name': 'airflow-test-pod',
> > 'image': 'ubuntu:16.04',
> > 'labels': {'foo': 'bar'},
> > 'namespace': 'default',
> > 'cmds': ['bash', '-cx'],
> > 'arguments': ['echo', '10'],
> > 'in_cluster': False,
> > 'get_logs': True
> >
> > I don't understand how these things are being passed via kwargs?
> >
> > --
> > Craig
> >
> >
> > On Wed, May 30, 2018 at 1:40 AM Driesprong, Fokko <fo...@driesprong.frl>
> > wrote:
> >
> > > Hi Craig,
> > >
> > > This is something that needs to be fixed. I agree with you this is very
> > > dirty. In your installation you're not installing the kubernetes stuff,
> > so
> > > the KubernetesPodOperator is ignored. We need to figure out how to have
> > > example dags that are not compatible with the vanilla installation, or
> we
> > > need to remove the kubernetes example for now, and move it to the
> > > documentation.
> > >
> > > Cheers, Fokko
> > >
> > >
> >
>
--
Craig Rodrigues
rodrigc@rodrigues.org
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by Chris Palmer <ch...@crpalmer.com>.
In the example
<https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_kubernetes_operator.py#L24>
it imports BaseOperator as KubernetesPodOperator, when the kubernetes
modules can't be found.
On Wed, May 30, 2018 at 3:34 PM, Craig Rodrigues <ro...@crodrigues.org>
wrote:
> For this particular DeprecationWarning, this problem is not caused
> by dependencies on kubernetes stuff.
>
> On this line:
> https://github.com/apache/incubator-airflow/blob/master/
> airflow/contrib/operators/kubernetes_pod_operator.py#L137
>
> The __init__() method for KubernetesPodOperator is calling the __init__()
> method
> for BaseOperator, and passing in values for kwargs that BaseOperator
> doesn't accept:
>
> name': 'airflow-test-pod',
> 'image': 'ubuntu:16.04',
> 'labels': {'foo': 'bar'},
> 'namespace': 'default',
> 'cmds': ['bash', '-cx'],
> 'arguments': ['echo', '10'],
> 'in_cluster': False,
> 'get_logs': True
>
> I don't understand how these things are being passed via kwargs?
>
> --
> Craig
>
>
> On Wed, May 30, 2018 at 1:40 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Hi Craig,
> >
> > This is something that needs to be fixed. I agree with you this is very
> > dirty. In your installation you're not installing the kubernetes stuff,
> so
> > the KubernetesPodOperator is ignored. We need to figure out how to have
> > example dags that are not compatible with the vanilla installation, or we
> > need to remove the kubernetes example for now, and move it to the
> > documentation.
> >
> > Cheers, Fokko
> >
> >
>
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by Craig Rodrigues <ro...@crodrigues.org>.
For this particular DeprecationWarning, this problem is not caused
by dependencies on kubernetes stuff.
On this line:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/kubernetes_pod_operator.py#L137
The __init__() method for KubernetesPodOperator is calling the __init__()
method
for BaseOperator, and passing in values for kwargs that BaseOperator
doesn't accept:
name': 'airflow-test-pod',
'image': 'ubuntu:16.04',
'labels': {'foo': 'bar'},
'namespace': 'default',
'cmds': ['bash', '-cx'],
'arguments': ['echo', '10'],
'in_cluster': False,
'get_logs': True
I don't understand how these things are being passed via kwargs?
--
Craig
On Wed, May 30, 2018 at 1:40 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Craig,
>
> This is something that needs to be fixed. I agree with you this is very
> dirty. In your installation you're not installing the kubernetes stuff, so
> the KubernetesPodOperator is ignored. We need to figure out how to have
> example dags that are not compatible with the vanilla installation, or we
> need to remove the kubernetes example for now, and move it to the
> documentation.
>
> Cheers, Fokko
>
>
Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Craig,
This is something that needs to be fixed. I agree with you this is very
dirty. In your installation you're not installing the kubernetes stuff, so
the KubernetesPodOperator is ignored. We need to figure out how to have
example dags that are not compatible with the vanilla installation, or we
need to remove the kubernetes example for now, and move it to the
documentation.
Cheers, Fokko
2018-05-30 2:11 GMT+02:00 Craig Rodrigues <cr...@gmail.com>:
> I tested master branch by putting the following in my requirements.txt:
>
> git+https://github.com/rodrigc/incubator-airflow@
> master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,
> ldap,mysql,postgres,redis,slack,s3]
>
> and did a pip install -r requirements.txt
>
> When I started the airflow webserver, I saw deprecation warnings. I
> put some additional debugging in models.py to through an exception so that
> I could see the
> full stacktrace:
>
> [2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import:
> /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/example_dags/example_kubernetes_operator.py
> Traceback (most recent call last):
> File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py",
> line 304, in process_file
> m = imp.load_source(mod_name, filepath)
> File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/example_dags/example_kubernetes_operator.py", line 53, in <module>
> dag=dag)
> File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/utils/decorators.py",
> line 98, in wrapper
> result = func(*args, **kwargs)
> File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py",
> line 2308, in __init__
> raise Exception("Invalid use of args or kwargs")
> Exception: Invalid use of args or kwargs
>
>
> If looks like example_kubernetes_operator.py, this code is the source of
> the exception:
>
> k = KubernetesPodOperator(
> namespace='default',
> image="ubuntu:16.04",
> cmds=["bash", "-cx"],
> arguments=["echo", "10"],
> labels={"foo": "bar"},
> name="airflow-test-pod",
> in_cluster=False,
> task_id="task",
> get_logs=True,
> dag=dag)
>
>
> Without my extra debugging, the deprecation warning looks like this:
>
> [2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING -
> Could not import KubernetesPodOperator
> /Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py:2315:
> PendingDeprecationWarning: Invalid arguments were passed to BaseOperator.
> Support for passing such arguments will be dropped in Airflow 2.0. Invalid
> arguments were:
> *args: ()
> **kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04', 'labels':
> {'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'],
> 'arguments': ['echo', '10'], 'in_cluster': False, 'get_logs': True}
> category=PendingDeprecationWarning
>
>
>
> What is the correct fix for this? It looks like a lot of operators pass
> in arguments which are not
> processed by BaseOperator, and thus trip over this deprecation warning.
>
> --
> Craig
>