You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/27 09:30:41 UTC
[GitHub] [airflow] opendatakraken opened a new issue #16111: Error reading command stream when executing git
opendatakraken opened a new issue #16111:
URL: https://github.com/apache/airflow/issues/16111
**Apache Airflow version**: 2.1.0
**Environment**:
- **OS**: CentOS 7
- **Kernel**: Linux app-tddo-pc1.fhm.de 3.10.0-1127.10.1.el7.x86_64 #1 SMP Wed Jun 3 14:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**: pip
- **Others**:
**What happened**:
I have several Pods with Alpine OS, python 3.8 and AirFlow 2.1.0 that work perfectly.
I have some DAGs that perform a `git clone <repo_url>` in a BashOperator and work well.
Now I have AirFlow 2.1.0 on a CentOS server (no VM) with Python 3.8 (but I tried both Python 3.6.x and 3.9.x)
The `git clone <repo_url>` doesn't work in this new installation. I always get the following error:
`Error reading command stream`
I tried several ways to perform the `git clone`:
- BashOperator
- PythonOperator using either the os or the subprocess libraries
- PythonOperator calling GitPython: git.Git("dir").clone("repo_url") (it executes a git clone command)
However, I can successfully perform the `git clone` in all the mentioned ways (os, subrocess and GitPython) from the python console on the same server and from the same user running AirFlow.
**What you expected to happen**:
`git clone` is successful
**How to reproduce it**:
Install AirFlow on CentOS7 and create a DAG that execute a `git clone` in a BashOperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849508328
No idea. But if it works interactively and does not work in non-interactive airflow, it's likely the configuration of your git, not Airflow. Try it with cron job for example and see if you have the same problem. Solve it there - it will be solved by airflow as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #16111:
URL: https://github.com/apache/airflow/issues/16111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849489341
Some relevant discussion that might help you https://stackoverflow.com/questions/23563062/how-do-i-force-git-not-to-prompt-for-credentials
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] opendatakraken commented on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
opendatakraken commented on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849648963
I tried with cron and it works. I also tried a `git pull` from the same DAG and it works. It looks like only git clone doesn't work.
But I managed to solve the issue: the delivered git 1.8.3.1 looks too old, after upgrading to 2.30.1 everything looks fine.
Interactive/non-interactive, passwords, etc. don't play any role cause git executed through the BashOperator can use user/passwd defined in ~/.git-credentials.
_There are plenty of reasons why such connectivity might fail, but they are environmental problem, not airflow._
Finally, it was the git version in CentOS7.
Regards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849489341
Some relevant discussions that might help you https://stackoverflow.com/questions/23563062/how-do-i-force-git-not-to-prompt-for-credentials
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849485637
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849488245
This likely comes with the way you have your authentication configured in Git. The error comes from transport helper which does authentication. If you are logged in interactively with your user it likely authenticates you using some of the native authentiction mechanisms of your system. Airflow workers do not work in interactive mode, so you have to make sure your authentication mechanisms work also in this mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849488245
This likely comes with the way you have your authentication configured for Git in your system.
\The error comes from transport helper which does authentication. If you are logged in interactively with your user it likely authenticates you using some of the native authentiction mechanisms of your system. Airflow workers do not work in interactive mode, so you have to make sure your authentication mechanisms work also in this mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849512630
One reason might be also that your host is not in "known-host" file - which happens when you first time connect to the host. Then it will ask you whether it's ok to add it (and exit on airflow because there is no-one to confirm it) this can be solved by adding the host key manually or using `-o StrictHostKeyChecking=no`.
Yet another reason might be that the server closes the connection for some reason (wrong IP/wrong proxy etc.). then you need to look for the reason in the server logs.
There are plenty of reasons why such connectivity might fail, but they are environmental problem, not airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] opendatakraken commented on issue #16111: Error reading command stream when executing git
Posted by GitBox <gi...@apache.org>.
opendatakraken commented on issue #16111:
URL: https://github.com/apache/airflow/issues/16111#issuecomment-849503163
But my git-url contains `username:password`:
https://username:password@mygitrepository/myproject.git
And why does it work in the alpineOS inside the pods but not on centos on the servers?
I didn't configure git credentials anywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org