You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/06 22:59:40 UTC
[GitHub] [airflow] SasanAhmadi opened a new pull request, #22802: bugfix for when polling for the created job, if fail to get job info it should not fail the task, instead it should continue polling until reaches the max allowed polling tries
SasanAhmadi opened a new pull request, #22802:
URL: https://github.com/apache/airflow/pull/22802
This is to eliminate a bug in jenkins job trigger opertor. During polling for the newly created build, it is possible to get a failure and it is causing the task to fail and if task have retries then it will result in duplicate builds which could lead to problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22802:
URL: https://github.com/apache/airflow/pull/22802#discussion_r844755417
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -153,9 +153,18 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
# once it will be available in python-jenkins (v > 0.4.15)
self.log.info('Polling jenkins queue at the url %s', location)
while try_count < self.max_try_before_job_appears:
- location_answer = jenkins_request_with_headers(
- jenkins_server, Request(method='POST', url=location)
- )
+ try:
+ location_answer = jenkins_request_with_headers(
+ jenkins_server, Request(method='POST', url=location)
+ )
+ # we don't want to fail the operator, this will continue to poll
+ # until max_try_before_job_appears reached
+ except (HTTPError, JenkinsException) as ex:
+ self.log.info(f'polling failed, retry polling. Failure reason: {ex}')
Review Comment:
Do we want to use `warning` here instead? (No preference personally.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SasanAhmadi commented on a diff in pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
SasanAhmadi commented on code in PR #22802:
URL: https://github.com/apache/airflow/pull/22802#discussion_r844792537
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -153,9 +153,18 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
# once it will be available in python-jenkins (v > 0.4.15)
self.log.info('Polling jenkins queue at the url %s', location)
while try_count < self.max_try_before_job_appears:
- location_answer = jenkins_request_with_headers(
- jenkins_server, Request(method='POST', url=location)
- )
+ try:
+ location_answer = jenkins_request_with_headers(
+ jenkins_server, Request(method='POST', url=location)
+ )
+ # we don't want to fail the operator, this will continue to poll
+ # until max_try_before_job_appears reached
+ except (HTTPError, JenkinsException) as ex:
+ self.log.info(f'polling failed, retry polling. Failure reason: {ex}')
Review Comment:
corrected, thanks for great comment!
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -168,6 +177,7 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
return build_number
try_count += 1
time.sleep(self.sleep_time)
+
raise AirflowException(
"The job hasn't been executed after polling " f"the queue {self.max_try_before_job_appears} times"
Review Comment:
applied
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22802:
URL: https://github.com/apache/airflow/pull/22802#discussion_r844586382
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -153,21 +153,30 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
# once it will be available in python-jenkins (v > 0.4.15)
self.log.info('Polling jenkins queue at the url %s', location)
while try_count < self.max_try_before_job_appears:
- location_answer = jenkins_request_with_headers(
- jenkins_server, Request(method='POST', url=location)
- )
- if location_answer is not None:
- json_response = json.loads(location_answer['body'])
- if (
- 'executable' in json_response
- and json_response['executable'] is not None
- and 'number' in json_response['executable']
- ):
- build_number = json_response['executable']['number']
- self.log.info('Job executed on Jenkins side with the build number %s', build_number)
- return build_number
- try_count += 1
- time.sleep(self.sleep_time)
+ try:
+ location_answer = jenkins_request_with_headers(
+ jenkins_server, Request(method='POST', url=location)
+ )
+ if location_answer is not None:
+ json_response = json.loads(location_answer['body'])
+ if (
+ 'executable' in json_response
+ and json_response['executable'] is not None
+ and 'number' in json_response['executable']
+ ):
+ build_number = json_response['executable']['number']
+ self.log.info('Job executed on Jenkins side with the build number %s', build_number)
+ return build_number
+ try_count += 1
+ time.sleep(self.sleep_time)
+
+ # we don't want to fail the operator, this will continue to poll
+ # until max_try_before_job_appears reached
+ except (HTTPError, JenkinsException) as ex:
Review Comment:
When are these exceptions raised, respectively? This try-catch block seems too big, a lot of the wrapped code is response processing that should be outside of this block, from what I can tell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #22802:
URL: https://github.com/apache/airflow/pull/22802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22802:
URL: https://github.com/apache/airflow/pull/22802#discussion_r844757522
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -168,6 +177,7 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
return build_number
try_count += 1
time.sleep(self.sleep_time)
+
raise AirflowException(
"The job hasn't been executed after polling " f"the queue {self.max_try_before_job_appears} times"
Review Comment:
While we’re changing this file, can you also fix this line too? It should be
```python
f"The job hasn't been executed after polling the queue {self.max_try_before_job_appears} times"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SasanAhmadi commented on a diff in pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
SasanAhmadi commented on code in PR #22802:
URL: https://github.com/apache/airflow/pull/22802#discussion_r844792631
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -168,6 +177,7 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
return build_number
try_count += 1
time.sleep(self.sleep_time)
+
raise AirflowException(
"The job hasn't been executed after polling " f"the queue {self.max_try_before_job_appears} times"
Review Comment:
applied
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SasanAhmadi commented on a diff in pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
SasanAhmadi commented on code in PR #22802:
URL: https://github.com/apache/airflow/pull/22802#discussion_r844620607
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -153,21 +153,30 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
# once it will be available in python-jenkins (v > 0.4.15)
self.log.info('Polling jenkins queue at the url %s', location)
while try_count < self.max_try_before_job_appears:
- location_answer = jenkins_request_with_headers(
- jenkins_server, Request(method='POST', url=location)
- )
- if location_answer is not None:
- json_response = json.loads(location_answer['body'])
- if (
- 'executable' in json_response
- and json_response['executable'] is not None
- and 'number' in json_response['executable']
- ):
- build_number = json_response['executable']['number']
- self.log.info('Job executed on Jenkins side with the build number %s', build_number)
- return build_number
- try_count += 1
- time.sleep(self.sleep_time)
+ try:
+ location_answer = jenkins_request_with_headers(
+ jenkins_server, Request(method='POST', url=location)
+ )
+ if location_answer is not None:
+ json_response = json.loads(location_answer['body'])
+ if (
+ 'executable' in json_response
+ and json_response['executable'] is not None
+ and 'number' in json_response['executable']
+ ):
+ build_number = json_response['executable']['number']
+ self.log.info('Job executed on Jenkins side with the build number %s', build_number)
+ return build_number
+ try_count += 1
+ time.sleep(self.sleep_time)
+
+ # we don't want to fail the operator, this will continue to poll
+ # until max_try_before_job_appears reached
+ except (HTTPError, JenkinsException) as ex:
Review Comment:
you are right it was a little careless to include everything in try except,
the only part raising exceptions is the call to `jenkins_request_with_headers` method,
I changed the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #22802:
URL: https://github.com/apache/airflow/pull/22802#discussion_r844756581
##########
airflow/providers/jenkins/operators/jenkins_job_trigger.py:
##########
@@ -153,9 +153,18 @@ def poll_job_in_queue(self, location: str, jenkins_server: Jenkins) -> int:
# once it will be available in python-jenkins (v > 0.4.15)
self.log.info('Polling jenkins queue at the url %s', location)
while try_count < self.max_try_before_job_appears:
- location_answer = jenkins_request_with_headers(
- jenkins_server, Request(method='POST', url=location)
- )
+ try:
+ location_answer = jenkins_request_with_headers(
+ jenkins_server, Request(method='POST', url=location)
+ )
+ # we don't want to fail the operator, this will continue to poll
+ # until max_try_before_job_appears reached
+ except (HTTPError, JenkinsException) as ex:
+ self.log.info(f'polling failed, retry polling. Failure reason: {ex}')
Review Comment:
Either way though, this should use `self.log.xxx("Polling failed, retrying.", exc_info=True)` instead. Don’t pass the exception in manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22802: Bug Fix for `apache-airflow-providers-jenkins` `JenkinsJobTriggerOperator`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22802:
URL: https://github.com/apache/airflow/pull/22802#issuecomment-1091194213
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org