You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/28 15:27:38 UTC
[GitHub] [airflow] nicolamarangoni opened a new issue, #25377: SSLCertVerificationError from JenkinsJobTriggerOperator
nicolamarangoni opened a new issue, #25377:
URL: https://github.com/apache/airflow/issues/25377
### Apache Airflow version
2.3.3 (latest released)
### What happened
After upgrading to alpine 3.16.1, Python 3.10 and AirFlow 2.3.3, the JenkinsJobTriggerOperator cannot connect to Jenkins because it cannot verify the SSL certificate:
```
[2022-07-28, 16:37:34 CEST] {base.py:68} INFO - Using connection ID 'jenkins' for task execution.
[2022-07-28, 16:37:34 CEST] {warnings.py:109} WARNING - /usr/lib/python3.10/site-packages/airflow/models/connection.py:294: DeprecationWarning: Encountered non-JSON in `extra` field for connection 'jenkins'. Support for non-JSON `extra` will be removed in Airflow 3.0
self._validate_extra(extra_val, self.conn_id)
[2022-07-28, 16:37:34 CEST] {jenkins.py:46} INFO - Trying to connect to [https://jenkins.pharos.pke.fhm.de:443](https://jenkins.pharos.pke.fhm.de/)
[2022-07-28, 16:37:34 CEST] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/airflow/providers/jenkins/operators/jenkins_job_trigger.py", line 197, in execute
jenkins_response = self.build_job(jenkins_server, self.parameters)
File "/usr/lib/python3.10/site-packages/airflow/providers/jenkins/operators/jenkins_job_trigger.py", line 133, in build_job
return jenkins_request_with_headers(jenkins_server, request)
File "/usr/lib/python3.10/site-packages/airflow/providers/jenkins/operators/jenkins_job_trigger.py", line 51, in jenkins_request_with_headers
response = jenkins_server.jenkins_request(req)
File "/usr/lib/python3.10/site-packages/jenkins/__init__.py", line 571, in jenkins_request
self._maybe_add_auth()
File "/usr/lib/python3.10/site-packages/jenkins/__init__.py", line 410, in _maybe_add_auth
raise JenkinsException(
jenkins.JenkinsException: Unable to authenticate with any scheme:
auth(kerberos) HTTPSConnectionPool(host='jenkins.pharos.pke.fhm.de', port=443): Max retries exceeded with url: /api/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
auth(basic) HTTPSConnectionPool(host='jenkins.pharos.pke.fhm.de', port=443): Max retries exceeded with url: /api/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
[2022-07-28, 16:37:34 CEST] {taskinstance.py:1415} INFO - Marking task as FAILED. dag_id=test_start_jenkins_job, task_id=jenkins_job, execution_date=20220728T123730, start_date=20220728T123733, end_date=20220728T123734
[2022-07-28, 16:37:34 CEST] {standard_task_runner.py:92} ERROR - Failed to execute job 38 for task jenkins_job (Unable to authenticate with any scheme:
auth(kerberos) HTTPSConnectionPool(host='jenkins.pharos.pke.fhm.de', port=443): Max retries exceeded with url: /api/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
auth(basic) HTTPSConnectionPool(host='jenkins.pharos.pke.fhm.de', port=443): Max retries exceeded with url: /api/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))); 107)
[2022-07-28, 16:37:34 CEST] {local_task_job.py:156} INFO - Task exited with return code 1
```
### What you think should happen instead
The JenkinsJobTriggerOperator should successfully connect to Jenkins over https
### How to reproduce
Write a DAG containing a task using the JenkinsJobTriggerOperator to connect to a Jenkins server over https
### Operating System
Alpine Linux 3.16.1
### Versions of Apache Airflow Providers
```
apache-airflow-providers-apache-hdfs==3.0.1
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.2.0
apache-airflow-providers-common-sql==1.0.0
apache-airflow-providers-datadog==3.0.0
apache-airflow-providers-exasol==2.1.3
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-jenkins==3.0.0
apache-airflow-providers-microsoft-mssql==3.1.0
apache-airflow-providers-odbc==3.1.0
apache-airflow-providers-oracle==3.1.0
apache-airflow-providers-postgres==5.1.0
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-slack==5.1.0
apache-airflow-providers-sqlite==3.1.0
apache-airflow-providers-ssh==3.1.0
```
### Deployment
Other 3rd-party Helm chart
### Deployment details
One Pod on Kubernetes containing the following containers
1 Container for the webserver service
1 Container for the scheduler service
1 Container for the dag-processor service
1 Container for the flower service
1 Container for the redis service
2 or 3 containers for the celery workers services
Due to a previous issue crashing the scheduler with the message UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS, we substitute scheduler_job.py with the file https://raw.githubusercontent.com/tanelk/airflow/a4b22932e5ac9c2b6f37c8c58345eee0f63cae09/airflow/jobs/scheduler_job.py.
### Anything else
The jenkins operator could previously successfully connect to Jenkins over http.
We build our image on an alpine base image provided by our kubernetes admins that already contains the company ssl-certificates needed to interact with the several company's web services
It looks like the new setup (Alpine 3.16.1 + python 3.10.x + AirFlow 2.3.3) isn't able to use the ssl certificates already provided in the os in the file `/etc/ssl/certs/ca-certificates.crt`.
However python 3.10.x is the suspect not AirFlow itself.
Is there a way to tell the operator to use the certificates in a specific file?
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nicolamarangoni commented on issue #25377: SSLCertVerificationError from JenkinsJobTriggerOperator
Posted by GitBox <gi...@apache.org>.
nicolamarangoni commented on issue #25377:
URL: https://github.com/apache/airflow/issues/25377#issuecomment-1199614854
Well, it was neither Alpine Linux nor Python 3.10.x. It was pip, but it is only a supposition.
Here what happened...
Some days ago I had to downgrade 2 providers because the new versions of them were buggy, hier the Dockerfile
```
...
RUN pip install --upgrade pip
...
RUN pip install --ignore-installed apache-airflow-providers-postgres==5.0.0
RUN pip install --ignore-installed apache-airflow-providers-oracle==3.1.0
...
```
Usually python uses the operating system location to load the ssl-certificates `/etc/ssl/certs/ca-certificates.crt,` where our company-certs are maintained by our kubernetes admin in theis base images (that we must use), however after the last builds python was using `/usr/lib/python3.10/site-packages/certifi/cacert.pem`, where our company-ssl-certs are missing.
It turned out that the `--ignore-installed` option is guilty, at least in the latest version of pip.
It is sufficient to have this option at least in a single pip command in any position in the install script, to make the python interpreter switch certs-file.
After removing that option (anyway useless in my build), python and consequently airflow keep using the right certs-file and the SSL issue is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25377: SSLCertVerificationError from JenkinsJobTriggerOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25377:
URL: https://github.com/apache/airflow/issues/25377#issuecomment-1198360046
This the problem of Alpine, not Airlfow. you need to make sure you have the right system certificates installed. There is nothing we can do about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25377: SSLCertVerificationError from JenkinsJobTriggerOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25377:
URL: https://github.com/apache/airflow/issues/25377#issuecomment-1198361749
There are plenty of similar issues (with ways to solve) if you google for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25377: SSLCertVerificationError from JenkinsJobTriggerOperator
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25377: SSLCertVerificationError from JenkinsJobTriggerOperator
URL: https://github.com/apache/airflow/issues/25377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25377: SSLCertVerificationError from JenkinsJobTriggerOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25377:
URL: https://github.com/apache/airflow/issues/25377#issuecomment-1202880719
Cool. Glad you solved it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org