You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/19 13:30:00 UTC
[GitHub] [airflow] masayuki038 opened a new issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
masayuki038 opened a new issue #19071:
URL: https://github.com/apache/airflow/issues/19071
### Apache Airflow Provider(s)
ssh
### Versions of Apache Airflow Providers
apache-airflow-providers-ssh 2.2.0
### Apache Airflow version
2.1.3
### Operating System
Ubuntu 20.04.2 LTS (Focal Fossa)
### Deployment
Other
### Deployment details
_No response_
### What happened
Specified `command` of SSHOperator to the return value of @task function, it raised AttributeError "'XComArg' object has no attribute 'startswith'".
- DAG
```
from airflow.decorators import dag, task
from airflow.providers.ssh.operators.ssh import SSHOperator
from airflow.utils.dates import days_ago
DEFAULT_ARGS = {"owner": "airflow"}
# [START example_dag_decorator_ssh_usage]
@dag(default_args=DEFAULT_ARGS, schedule_interval=None, start_date=days_ago(2), tags=['example'])
def example_dag_decorator_ssh():
@task
def build_command() -> str:
return 'whoami'
command = build_command()
SSHOperator(
task_id='run_my_command',
command=command,
ssh_conn_id='default_conn',
retries=0
)
dag = example_dag_decorator_ssh()
# [END example_dag_decorator_ssh_usage]
```
- Test
```
import os
import unittest
from glob import glob
from airflow.models import DagBag
ROOT_FOLDER = os.path.realpath(
os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, os.pardir)
)
class TestExampleDagDecoratorSsh(unittest.TestCase):
def test_should_be_importable(self):
dagbag = DagBag(
dag_folder=f"{ROOT_FOLDER}/airflow/example_dags",
include_examples=False,
)
assert 0 == len(dagbag.import_errors), f"import_errors={str(dagbag.import_errors)}"
```
- Result
```
------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------
INFO airflow.models.dagbag.DagBag:dagbag.py:496 Filling up the DagBag from /home/masayuki/git/airflow/airflow/example_dags
ERROR airflow.models.dagbag.DagBag:dagbag.py:329 Failed to import: /home/masayuki/git/airflow/airflow/example_dags/example_dag_decorator_ssh.py
Traceback (most recent call last):
File "/home/masayuki/git/airflow/airflow/models/dagbag.py", line 326, in _load_modules_from_file
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/masayuki/git/airflow/airflow/example_dags/example_dag_decorator_ssh.py", line 26, in <module>
dag = example_dag_decorator_ssh()
File "/home/masayuki/git/airflow/airflow/models/dag.py", line 2351, in factory
f(**f_kwargs)
File "/home/masayuki/git/airflow/airflow/example_dags/example_dag_decorator_ssh.py", line 18, in example_dag_decorator_ssh
SSHOperator(
File "/home/masayuki/git/airflow/airflow/models/baseoperator.py", line 178, in apply_defaults
result = func(self, *args, **kwargs)
File "/home/masayuki/git/airflow/airflow/providers/ssh/operators/ssh.py", line 81, in __init__
self.get_pty = (self.command.startswith('sudo') or get_pty) if self.command else get_pty
AttributeError: 'XComArg' object has no attribute 'startswith'
```
### What you expected to happen
No error
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955626497
> @josh-fell Thanks for your updates 😄
>
> > In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
>
> Although I tried to pass the value wrapped with `str()` to SSHOperator, the upstream task of it is not set. Do you know the workaround ?
Have you tried setting the dependencies in the classic way? Something like:
```python
command = build_command()
ssh = SSHOperator(
task_id='run_my_command',
command=str(command),
ssh_conn_id='default_conn',
retries=0
)
command >> ssh
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] masayuki038 edited a comment on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
masayuki038 edited a comment on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955624703
@josh-fell Thanks for your updates 😄
> In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
Although I tried to pass the value wrapped with `str()` a SSHOperator, the upstream task of it is not set. Do you know the workaround ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955057129
Just in time for the next provider's release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946888333
@masayuki038 @ephraimbuddy Happy to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19071:
URL: https://github.com/apache/airflow/issues/19071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946725530
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] masayuki038 commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
masayuki038 commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946851990
@earthmancash2 EmailOperator looks like to be able to take XComArg as parameter. Why can't SSHOperator as well ?
https://github.com/apache/airflow/blob/829f19f94ea88db1825261ef2fd4920f8a9558eb/airflow/example_dags/example_dag_decorator.py#L61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955030428
@masayuki038 #19323 has been submitted to fix the issue. In the meantime, can you try wrapping the `command` arg in `str()` and see if that is an acceptable workaround for you?
```python
SSHOperator(task_id='run_my_command', command=str(command), ssh_conn_id='default_conn', retries=0)
```
The `XComArg` class has a string representation that mimics the classic `"{{ task_instance.xcom_pull(...) }}"` which may help in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] masayuki038 commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
masayuki038 commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955624703
@josh-fell Thanks for your updates 😄
> In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
Although I tried to pass the value wrapped with `str()` a SSHOperator, the upstream task of it is not set. Do you know the workaround ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946857144
The example you referenced is using the `.output` but you are passing the xcomarg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy removed a comment on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
ephraimbuddy removed a comment on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946857144
The example you referenced is using the `.output` but you are passing the xcomarg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946820350
You can't get xcom passed to the SSHOperator this way. What you want is only possible when both tasks are taskflow API.
For your case, you have to explicitly push xcom and use template strings to pull in SSHOperator.
```python
@task
def build_command(ti=None) -> str:
xcom = 'whoami'
ti.xcom_push(value=xcom)
ssh = SSHOperator(
task_id='run_my_command',
command="{{ ti.xcom_pull(task_ids=['build_command']) }}",
ssh_conn_id='default_conn',
retries=0
)
build_command >> ssh
```
Closing this as it's not an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946887052
Assigning it to @josh-fell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] masayuki038 edited a comment on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
masayuki038 edited a comment on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955624703
@josh-fell Thanks for your updates 😄
> In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
Although I tried to pass the value wrapped with `str()` to SSHOperator, the upstream task of it is not set. Do you know the workaround ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #19071:
URL: https://github.com/apache/airflow/issues/19071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946858939
Hmm. Reopening to investigate more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org