You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/28 05:59:21 UTC
[GitHub] [airflow] tandelDipak opened a new issue #13942: Bash Operator can't run custom bash commands
tandelDipak opened a new issue #13942:
URL: https://github.com/apache/airflow/issues/13942
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
<!-- A short description of your feature -->
I have defined one shell script function and in my .bashrc profile, I have added a line to source the bash script so that I can run the function defined inside the script. The same command doesn't work in Bash Operator.
**Use case / motivation**
<!-- What do you want to happen?
Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.
-->
I have to run multiples command (5-6 commands to run one script) using && operator to setup environment and other things. I have a custom command which takes 3 arguments and does all the work of 5-6 commands. This feature will help everyone to run custom commands to simplify the process.
I have done something similar to the below post. Those custom commands are not working in BashOperator.
https://medium.com/devnetwork/how-to-create-your-own-custom-terminal-commands-c5008782a78e
**Are you willing to submit a PR?**
<!--- We accept contributions! -->
Yes, I am willing to submit PR.
**Related Issues**
<!-- Is there currently another issue associated with this? -->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ulisesojeda commented on issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
ulisesojeda commented on issue #13942:
URL: https://github.com/apache/airflow/issues/13942#issuecomment-971818394
Hi @tandelDipak,
In order to define custom commands (functions) you can define it in a single line before calling.
The BashOperator is not defined to execute a script before running the **bash_command**
**Example:**
```
run_this = BashOperator(
dag=dag,
task_id='run_bash',
bash_command="print_my_input() { echo Your input: $1 ; } && print_my_input 222",
)
```
**Result:**
{subprocess.py:78} INFO - Your input: 222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13942:
URL: https://github.com/apache/airflow/issues/13942#issuecomment-971826977
This is not a problem with BashOperator, but misunderstanding how .bashrc works. The `.bashrc` is only sourced for "interactive" login and remote non-interactive shell sessions (executed via ssh-daemon). Baasically .bashrc will only be sourced automatically if there is a human typing the commands to execute.
For any kind of "unsupervised" bash command (for example when you run cron scripts) you need to manually source the .bashrc file if you want to process any custom code there:
```
source /home/airflow/.bashrc && command
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13942:
URL: https://github.com/apache/airflow/issues/13942#issuecomment-971826977
This is not a problem with BashOperator, but misunderstanding how .bashrc works. The `.bashrc` is only sourced for "interactive" login and remote non-interactive shell sessions (executed via ssh-daemon). Baasically .bashrc will only be sourced automatically if there is a human typing the command to execute.
For any kind of "unsupervised" bash command (for example when you run cron scripts) you need to manually source the .bashrc file if you want to process any custom code there:
```
source /opt/airflow/.bashrc && command
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #13942:
URL: https://github.com/apache/airflow/issues/13942#issuecomment-972577812
Actually `.bashrc` is only sourced for interactive _non-login_ shells; login shells read `.profile` and `.bash_profile`. But a lot of `.bash_profile` implementations automatically source `.bashrc` so, uh it’s complicated and off-topic anyway. I’ll just drop a link in case anyone is actually interested in the exact differences.
https://linuxize.com/post/bashrc-vs-bash-profile/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13942:
URL: https://github.com/apache/airflow/issues/13942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13942:
URL: https://github.com/apache/airflow/issues/13942#issuecomment-973198779
Oh yeah. It is :exploding_head: especially that for example MacOS has their own rules on which ones are loaded when as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #13942:
URL: https://github.com/apache/airflow/issues/13942#issuecomment-812595600
@tandelDipak you can always open PR directly, there is no need to open an issue first
feel free to submit your PR :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13942: Bash Operator can't run custom bash commands
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13942:
URL: https://github.com/apache/airflow/issues/13942#issuecomment-971826977
This is not a problem with BashOperator, but misunderstanding how .bashrc works. The `.bashrc` is only sourced for "interactive" login and remote non-interactive shell sessions (executed via ssh-daemon). Baasically .bashrc will only be sourced automatically if there is a human typing the commands to execute.
For any kind of "unsupervised" bash command (for example when you run cron scripts) you need to manually source the .bashrc file if you want to process any custom code there:
```
source /opt/airflow/.bashrc && command
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org