You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/19 13:30:00 UTC

[GitHub] [airflow] masayuki038 opened a new issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

masayuki038 opened a new issue #19071:
URL: https://github.com/apache/airflow/issues/19071


   ### Apache Airflow Provider(s)
   
   ssh
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-ssh 2.2.0
   
   ### Apache Airflow version
   
   2.1.3
   
   ### Operating System
   
   Ubuntu 20.04.2 LTS (Focal Fossa)
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   Specified `command` of SSHOperator to the return value of @task function, it raised AttributeError "'XComArg' object has no attribute 'startswith'".
   
   - DAG
   ```
   from airflow.decorators import dag, task
   from airflow.providers.ssh.operators.ssh import SSHOperator
   from airflow.utils.dates import days_ago
   
   DEFAULT_ARGS = {"owner": "airflow"}
   
   
   # [START example_dag_decorator_ssh_usage]
   @dag(default_args=DEFAULT_ARGS, schedule_interval=None, start_date=days_ago(2), tags=['example'])
   def example_dag_decorator_ssh():
   
       @task
       def build_command() -> str:
           return 'whoami'
   
       command = build_command()
   
       SSHOperator(
           task_id='run_my_command',
           command=command,
           ssh_conn_id='default_conn',
           retries=0
       )
   
   
   dag = example_dag_decorator_ssh()
   # [END example_dag_decorator_ssh_usage]
   ```
   
   - Test
   ```
   import os
   import unittest
   from glob import glob
   
   from airflow.models import DagBag
   
   ROOT_FOLDER = os.path.realpath(
       os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, os.pardir)
   )
   
   
   class TestExampleDagDecoratorSsh(unittest.TestCase):
       def test_should_be_importable(self):
           dagbag = DagBag(
               dag_folder=f"{ROOT_FOLDER}/airflow/example_dags",
               include_examples=False,
           )
           assert 0 == len(dagbag.import_errors), f"import_errors={str(dagbag.import_errors)}"
   
   ```
   
   - Result
   ```
   ------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------
   INFO     airflow.models.dagbag.DagBag:dagbag.py:496 Filling up the DagBag from /home/masayuki/git/airflow/airflow/example_dags
   ERROR    airflow.models.dagbag.DagBag:dagbag.py:329 Failed to import: /home/masayuki/git/airflow/airflow/example_dags/example_dag_decorator_ssh.py
   Traceback (most recent call last):
     File "/home/masayuki/git/airflow/airflow/models/dagbag.py", line 326, in _load_modules_from_file
       loader.exec_module(new_module)
     File "<frozen importlib._bootstrap_external>", line 850, in exec_module
     File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
     File "/home/masayuki/git/airflow/airflow/example_dags/example_dag_decorator_ssh.py", line 26, in <module>
       dag = example_dag_decorator_ssh()
     File "/home/masayuki/git/airflow/airflow/models/dag.py", line 2351, in factory
       f(**f_kwargs)
     File "/home/masayuki/git/airflow/airflow/example_dags/example_dag_decorator_ssh.py", line 18, in example_dag_decorator_ssh
       SSHOperator(
     File "/home/masayuki/git/airflow/airflow/models/baseoperator.py", line 178, in apply_defaults
       result = func(self, *args, **kwargs)
     File "/home/masayuki/git/airflow/airflow/providers/ssh/operators/ssh.py", line 81, in __init__
       self.get_pty = (self.command.startswith('sudo') or get_pty) if self.command else get_pty
   AttributeError: 'XComArg' object has no attribute 'startswith'
   ```
   
   ### What you expected to happen
   
   No error
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] josh-fell commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955626497


   > @josh-fell Thanks for your updates 😄
   > 
   > > In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
   > 
   > Although I tried to pass the value wrapped with `str()` to SSHOperator, the upstream task of it is not set. Do you know the workaround ?
   
   Have you tried setting the dependencies in the classic way? Something like:
   ```python
       command = build_command()
   
       ssh = SSHOperator(
           task_id='run_my_command',
           command=str(command),
           ssh_conn_id='default_conn',
           retries=0
       )
   
       command >> ssh
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] masayuki038 edited a comment on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
masayuki038 edited a comment on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955624703


   @josh-fell Thanks for your updates 😄 
   
   > In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
   
   Although I tried to pass the value wrapped with `str()` a SSHOperator, the upstream task of it is not set. Do you know the workaround ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955057129


   Just in time for the next provider's release
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] josh-fell commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946888333


   @masayuki038 @ephraimbuddy Happy to take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk closed issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19071:
URL: https://github.com/apache/airflow/issues/19071


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946725530


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] masayuki038 commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
masayuki038 commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946851990


   @earthmancash2 EmailOperator looks like to be able to take XComArg as parameter. Why can't SSHOperator as well ?
   https://github.com/apache/airflow/blob/829f19f94ea88db1825261ef2fd4920f8a9558eb/airflow/example_dags/example_dag_decorator.py#L61


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] josh-fell commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
josh-fell commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955030428


   @masayuki038 #19323 has been submitted to fix the issue. In the meantime, can you try wrapping the `command` arg in `str()` and see if that is an acceptable workaround for you?
   
   ```python
   SSHOperator(task_id='run_my_command', command=str(command), ssh_conn_id='default_conn', retries=0)
   ```
   
   The `XComArg` class has a string representation that mimics the classic `"{{ task_instance.xcom_pull(...) }}"` which may help in this case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] masayuki038 commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
masayuki038 commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955624703


   @josh-fell Thanks for your updates 😄 
   
   > In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
   Although I tried to pass the value wrapped with `str()` a SSHOperator, the upstream task of it is not set. Do you know the workaround ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946857144


   The example you referenced is using the `.output` but you are passing the xcomarg. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy removed a comment on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
ephraimbuddy removed a comment on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946857144


   The example you referenced is using the `.output` but you are passing the xcomarg. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946820350


   You can't get xcom passed to the SSHOperator this way. What you want is only possible when both tasks are taskflow API. 
   
   For your case, you have to explicitly push xcom and use template strings to pull in SSHOperator. 
   
   ```python
   
       @task
       def build_command(ti=None) -> str:
           xcom = 'whoami'
           ti.xcom_push(value=xcom)
   
       ssh = SSHOperator(
           task_id='run_my_command',
           command="{{ ti.xcom_pull(task_ids=['build_command']) }}",
           ssh_conn_id='default_conn',
           retries=0
       )
       build_command >> ssh
   ```
   
   Closing this as it's not an issue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946887052


   Assigning it to @josh-fell 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] masayuki038 edited a comment on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
masayuki038 edited a comment on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-955624703


   @josh-fell Thanks for your updates 😄 
   
   > In the meantime, can you try wrapping the command arg in str() and see if that is an acceptable workaround for you?
   
   Although I tried to pass the value wrapped with `str()` to SSHOperator, the upstream task of it is not set. Do you know the workaround ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy closed issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #19071:
URL: https://github.com/apache/airflow/issues/19071


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on issue #19071: Error in SSHOperator "'XComArg' object has no attribute 'startswith'"

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19071:
URL: https://github.com/apache/airflow/issues/19071#issuecomment-946858939


   Hmm. Reopening to investigate more


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org