You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@gmail.com> on 2019/07/19 03:24:04 UTC
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Hi all,
Seems this issue is re-happening again. Seems the PR link is properly
created in the corresponding JIRA but it doesn't change the JIRA's status
from OPEN to IN-PROGRESS.
See, for instance,
https://issues.apache.org/jira/browse/SPARK-28443
https://issues.apache.org/jira/browse/SPARK-28440
https://issues.apache.org/jira/browse/SPARK-28436
https://issues.apache.org/jira/browse/SPARK-28434
https://issues.apache.org/jira/browse/SPARK-28433
https://issues.apache.org/jira/browse/SPARK-28431
Josh and Dongjoon, do you guys maybe have any idea?
2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> Thank you so much Josh .. !!
>
> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>
>> The code for this runs in http://spark-prs.appspot.com (see
>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>> )
>>
>> I checked the AppEngine logs and it looks like we're getting error
>> responses, possibly due to a credentials issue:
>>
>> Exception when starting progress on JIRA issue SPARK-27355 (
>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>> Traceback (most recent call last): File
>>> Traceback (most recent call last):
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>> line 138
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>> issue_number)) File
>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>> issue_number))
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>> line 27
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in start_issue_progress jira_client = get_jira_client() File
>>> jira_client = get_jira_client()
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>> line 18
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>> app.config['JIRA_PASSWORD']))
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>> line 472
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in __init__ si = self.server_info() File
>>> si = self.server_info()
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>> line 2133
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in server_info j = self._get_json('serverInfo') File
>>> j = self._get_json('serverInfo')
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>> line 2549
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in _get_json r = self._session.get(url, params=params) File
>>> r = self._session.get(url, params=params)
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>> line 151
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in get return self.__verb('GET', url, **kwargs) File
>>> return self.__verb('GET', url, **kwargs)
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>> line 147
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>> raise_on_error(response, verb=verb, **kwargs)
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>> line 57
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>> CAPTCHA_CHALLENGE; login-url=https://issues.apache.org/jira/login.jsp r.status_code,
>>> error, r.url, request=request, response=r, **kwargs)
>>> JIRAError: JiraError HTTP 403 url:
>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>> text: CAPTCHA_CHALLENGE; login-url=
>>> https://issues.apache.org/jira/login.jsp
>>
>>
>> It looks like ASF JIRA was throwing a captcha challenge at us, so I used
>> the credentials to manually log in and complete the challenge.
>>
>> Hopefully that's enough to fix things, but to prevent re-occurrence we
>> might need to change the login credential type from username + password to
>> instead use OAuth tokens.
>>
>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>> wrote:
>>
>>> Can anyone take a look for this one? OPEN status JIRAs are being rapidly
>>> increased (from around 2400 to 2600)
>>>
>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>
>>>> Hi all,
>>>>
>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>> somewhere.
>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>> somebody opens a PR against a JIRA.
>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>
>>>> Can someone else who knows where it's running can check this?
>>>>
>>>> FWIW, I check every PR and JIRA almost every day but ever since this
>>>> happened, this makes (at least to me) duplicately check the JIRAs.
>>>> Previously, if I check all the PRs and JIRAs, they were not duplicated
>>>> because JIRAs having PRs have different status, "IN PROGRESS" but now all
>>>> JIRAs have "OPEN" status.
>>>>
>>>> Thanks.
>>>>
>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Josh Rosen <ro...@gmail.com>.
(Catching up on a backlog of emails, hence my belated reply)
I just checked the spark-prs app engine logs and it appears that our JIRA
API calls are failing due to a CAPTCHA check (same issue as before). Based
on https://jira.atlassian.com/browse/JRASERVER-40362, it sounds like this
is a fairly common problem.
I manually logged into the 'apachespark' JIRA account and completed the
CPATCHA, so *hopefully* things should be temporarily unbroken.
To permanently fix this issue, we might need to use OAuth tokens for
connecting to JIRA (instead of basic username + password auth). It looks
like the Python JIRA library supports this (
https://jira.readthedocs.io/en/master/examples.html#oauth) and I found some
promising-looking instructions on how to generate the OAuth tokens:
https://www.redradishtech.com/display/KB/How+to+write+a+Python+script+authenticating+with+Jira+via+OAuth
.
However, it looks like you need to be a JIRA administrator in order to
configure the applink so I can't fix this by myself.
It would be great if the native GitHub <-> JIRA integration could meet our
needs; this probably either didn't exist or wasn't configurable by us when
we first wrote our own integration / sync script.
On Wed, Apr 29, 2020 at 6:21 PM Hyukjin Kwon <gu...@gmail.com> wrote:
> Let actually me just take a look by myself and bring some updates soon.
>
> 2020년 4월 30일 (목) 오전 9:13, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> WDYT @Josh Rosen <jo...@databricks.com>?
>> Seems
>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L131-L142 this
>> isn't working anymore.
>> Does it make sense to move it to native Jira-GitHub integration
>> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>
>> ?
>> It won't change JIRA status as we used to do but it might be better from
>> a cursory look. However, maybe I missed some context.
>>
>>
>> 2020년 4월 30일 (목) 오전 2:46, Nicholas Chammas <ni...@gmail.com>님이
>> 작성:
>>
>>> Not sure what you mean. The native integration will auto-link from a
>>> Jira ticket to the PRs that mention that ticket. I don't think it will
>>> update the ticket's status, though.
>>>
>>> Would you like me to file a ticket with Infra and see what they say?
>>>
>>> On Tue, Apr 28, 2020 at 12:21 AM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> Maybe it's time to switch. Do you know if we can still link the JIRA
>>>> against Github?
>>>> The script used to change the status of JIRA too but it stopped working
>>>> for a long time - I suspect this isn't a big deal.
>>>>
>>>> 2020년 4월 25일 (토) 오전 10:31, Nicholas Chammas <ni...@gmail.com>님이
>>>> 작성:
>>>>
>>>>> Have we asked Infra recently about enabling the native Jira-GitHub
>>>>> integration
>>>>> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>?
>>>>> Maybe we can deprecate the part of this script that updates Jira tickets
>>>>> with links to the PR and rely on the native integration instead. We use it
>>>>> at my day job, for example.
>>>>>
>>>>> On Fri, Apr 24, 2020 at 12:39 AM Hyukjin Kwon <gu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Seems like this github_jira_sync.py
>>>>>> <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
>>>>>> seems stopped working completely now.
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-31532 <>
>>>>>> https://github.com/apache/spark/pull/28316
>>>>>> https://issues.apache.org/jira/browse/SPARK-31529 <>
>>>>>> https://github.com/apache/spark/pull/28315
>>>>>> https://issues.apache.org/jira/browse/SPARK-31528 <>
>>>>>> https://github.com/apache/spark/pull/28313
>>>>>>
>>>>>> Josh, would you mind taking a look please when you find some time?
>>>>>> There is a bunch of JIRAs now, and it is very confusing which JIRA is
>>>>>> in progress with a PR or not.
>>>>>>
>>>>>>
>>>>>> 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>
>>>>>>> Just FYI, I had to come up with a better JQL to filter out the JIRAs
>>>>>>> that already have linked PRs.
>>>>>>> In case it helps someone, I use this JQL now to look through the
>>>>>>> open JIRAs:
>>>>>>>
>>>>>>> project = SPARK AND
>>>>>>> status = Open AND
>>>>>>> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
>>>>>>> ORDER BY created DESC, priority DESC, updated DESC
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>>
>>>>>>>> That's a great explanation. Thanks I didn't know that.
>>>>>>>>
>>>>>>>> Josh, do you know who I should ping on this?
>>>>>>>>
>>>>>>>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi, Hyukjin.
>>>>>>>>>
>>>>>>>>> In short, there are two bots. And, the current situation happens
>>>>>>>>> when only one bot with `dev/github_jira_sync.py` works.
>>>>>>>>>
>>>>>>>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status
>>>>>>>>> change because it only use `add_remote_link` and `add_comment` API.
>>>>>>>>> I know only this bot (in Apache Spark repository repo)
>>>>>>>>>
>>>>>>>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>>>>>>>> `githubbot` (Name: `ASF GitHub Bot`).
>>>>>>>>> And, the other bot's activity is done under JIRA ID `apachespark`
>>>>>>>>> (Name: `Apache Spark`).
>>>>>>>>> The other bot is the one which Josh mentioned before. (in
>>>>>>>>> `databricks/spark-pr-dashboard` repo).
>>>>>>>>>
>>>>>>>>> The root cause will be the same. The API key used by the bot is
>>>>>>>>> rejected by Apache JIRA and forwarded to CAPCHAR.
>>>>>>>>>
>>>>>>>>> Bests,
>>>>>>>>> Dongjoon.
>>>>>>>>>
>>>>>>>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Seems this issue is re-happening again. Seems the PR link is
>>>>>>>>>> properly created in the corresponding JIRA but it doesn't change the JIRA's
>>>>>>>>>> status from OPEN to IN-PROGRESS.
>>>>>>>>>>
>>>>>>>>>> See, for instance,
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>>>>>>>
>>>>>>>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>>>>>>>
>>>>>>>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이
>>>>>>>>>> 작성:
>>>>>>>>>>
>>>>>>>>>>> Thank you so much Josh .. !!
>>>>>>>>>>>
>>>>>>>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이
>>>>>>>>>>> 작성:
>>>>>>>>>>>
>>>>>>>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>>>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>>>>>>>> )
>>>>>>>>>>>>
>>>>>>>>>>>> I checked the AppEngine logs and it looks like we're getting
>>>>>>>>>>>> error responses, possibly due to a credentials issue:
>>>>>>>>>>>>
>>>>>>>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>>>>>>>> Traceback (most recent call last): File
>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>>>>>>>> line 138
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>>>>> issue_number)) File
>>>>>>>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>>>>> issue_number))
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>>>>> line 27
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>>>>>>>> jira_client = get_jira_client()
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>>>>> line 18
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>>>> line 472
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in __init__ si = self.server_info() File
>>>>>>>>>>>>> si = self.server_info()
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>>>> line 2133
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>>>>>>>> j = self._get_json('serverInfo')
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>>>> line 2549
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>>>>>>>> r = self._session.get(url, params=params)
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>>>> line 151
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>>>> line 147
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>>>> line 57
>>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>>>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code,
>>>>>>>>>>>>> error, r.url, request=request, response=r, **kwargs)
>>>>>>>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us,
>>>>>>>>>>>> so I used the credentials to manually log in and complete the challenge.
>>>>>>>>>>>>
>>>>>>>>>>>> Hopefully that's enough to fix things, but to prevent
>>>>>>>>>>>> re-occurrence we might need to change the login credential type from
>>>>>>>>>>>> username + password to instead use OAuth tokens.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <
>>>>>>>>>>>> gurwls223@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are
>>>>>>>>>>>>> being rapidly increased (from around 2400 to 2600)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이
>>>>>>>>>>>>> 작성:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running
>>>>>>>>>>>>>> correctly somewhere.
>>>>>>>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS"
>>>>>>>>>>>>>> when
>>>>>>>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>>>>>>>> Looks now it only leaves a link and does not change JIRA's
>>>>>>>>>>>>>> status.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever
>>>>>>>>>>>>>> since this happened, this makes (at least to me) duplicately check the
>>>>>>>>>>>>>> JIRAs.
>>>>>>>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Hyukjin Kwon <gu...@gmail.com>.
Let actually me just take a look by myself and bring some updates soon.
2020년 4월 30일 (목) 오전 9:13, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> WDYT @Josh Rosen <jo...@databricks.com>?
> Seems
> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L131-L142 this
> isn't working anymore.
> Does it make sense to move it to native Jira-GitHub integration
> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>
> ?
> It won't change JIRA status as we used to do but it might be better from a
> cursory look. However, maybe I missed some context.
>
>
> 2020년 4월 30일 (목) 오전 2:46, Nicholas Chammas <ni...@gmail.com>님이
> 작성:
>
>> Not sure what you mean. The native integration will auto-link from a Jira
>> ticket to the PRs that mention that ticket. I don't think it will update
>> the ticket's status, though.
>>
>> Would you like me to file a ticket with Infra and see what they say?
>>
>> On Tue, Apr 28, 2020 at 12:21 AM Hyukjin Kwon <gu...@gmail.com>
>> wrote:
>>
>>> Maybe it's time to switch. Do you know if we can still link the JIRA
>>> against Github?
>>> The script used to change the status of JIRA too but it stopped working
>>> for a long time - I suspect this isn't a big deal.
>>>
>>> 2020년 4월 25일 (토) 오전 10:31, Nicholas Chammas <ni...@gmail.com>님이
>>> 작성:
>>>
>>>> Have we asked Infra recently about enabling the native Jira-GitHub
>>>> integration
>>>> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>?
>>>> Maybe we can deprecate the part of this script that updates Jira tickets
>>>> with links to the PR and rely on the native integration instead. We use it
>>>> at my day job, for example.
>>>>
>>>> On Fri, Apr 24, 2020 at 12:39 AM Hyukjin Kwon <gu...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Seems like this github_jira_sync.py
>>>>> <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
>>>>> seems stopped working completely now.
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-31532 <>
>>>>> https://github.com/apache/spark/pull/28316
>>>>> https://issues.apache.org/jira/browse/SPARK-31529 <>
>>>>> https://github.com/apache/spark/pull/28315
>>>>> https://issues.apache.org/jira/browse/SPARK-31528 <>
>>>>> https://github.com/apache/spark/pull/28313
>>>>>
>>>>> Josh, would you mind taking a look please when you find some time?
>>>>> There is a bunch of JIRAs now, and it is very confusing which JIRA is
>>>>> in progress with a PR or not.
>>>>>
>>>>>
>>>>> 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>
>>>>>> Just FYI, I had to come up with a better JQL to filter out the JIRAs
>>>>>> that already have linked PRs.
>>>>>> In case it helps someone, I use this JQL now to look through the open
>>>>>> JIRAs:
>>>>>>
>>>>>> project = SPARK AND
>>>>>> status = Open AND
>>>>>> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
>>>>>> ORDER BY created DESC, priority DESC, updated DESC
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>
>>>>>>> That's a great explanation. Thanks I didn't know that.
>>>>>>>
>>>>>>> Josh, do you know who I should ping on this?
>>>>>>>
>>>>>>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, Hyukjin.
>>>>>>>>
>>>>>>>> In short, there are two bots. And, the current situation happens
>>>>>>>> when only one bot with `dev/github_jira_sync.py` works.
>>>>>>>>
>>>>>>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status
>>>>>>>> change because it only use `add_remote_link` and `add_comment` API.
>>>>>>>> I know only this bot (in Apache Spark repository repo)
>>>>>>>>
>>>>>>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>>>>>>> `githubbot` (Name: `ASF GitHub Bot`).
>>>>>>>> And, the other bot's activity is done under JIRA ID `apachespark`
>>>>>>>> (Name: `Apache Spark`).
>>>>>>>> The other bot is the one which Josh mentioned before. (in
>>>>>>>> `databricks/spark-pr-dashboard` repo).
>>>>>>>>
>>>>>>>> The root cause will be the same. The API key used by the bot is
>>>>>>>> rejected by Apache JIRA and forwarded to CAPCHAR.
>>>>>>>>
>>>>>>>> Bests,
>>>>>>>> Dongjoon.
>>>>>>>>
>>>>>>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Seems this issue is re-happening again. Seems the PR link is
>>>>>>>>> properly created in the corresponding JIRA but it doesn't change the JIRA's
>>>>>>>>> status from OPEN to IN-PROGRESS.
>>>>>>>>>
>>>>>>>>> See, for instance,
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>>>>>>
>>>>>>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>>>>>>
>>>>>>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>>>>
>>>>>>>>>> Thank you so much Josh .. !!
>>>>>>>>>>
>>>>>>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>>>>>>>>
>>>>>>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>> I checked the AppEngine logs and it looks like we're getting
>>>>>>>>>>> error responses, possibly due to a credentials issue:
>>>>>>>>>>>
>>>>>>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>>>>>>> Traceback (most recent call last): File
>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>>>>>>> line 138
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>>>> issue_number)) File
>>>>>>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>>>> issue_number))
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>>>> line 27
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>>>>>>> jira_client = get_jira_client()
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>>>> line 18
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>>> line 472
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in __init__ si = self.server_info() File
>>>>>>>>>>>> si = self.server_info()
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>>> line 2133
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>>>>>>> j = self._get_json('serverInfo')
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>>> line 2549
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>>>>>>> r = self._session.get(url, params=params)
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>>> line 151
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>>> line 147
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>>> line 57
>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error,
>>>>>>>>>>>> r.url, request=request, response=r, **kwargs)
>>>>>>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us,
>>>>>>>>>>> so I used the credentials to manually log in and complete the challenge.
>>>>>>>>>>>
>>>>>>>>>>> Hopefully that's enough to fix things, but to prevent
>>>>>>>>>>> re-occurrence we might need to change the login credential type from
>>>>>>>>>>> username + password to instead use OAuth tokens.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <
>>>>>>>>>>> gurwls223@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are
>>>>>>>>>>>> being rapidly increased (from around 2400 to 2600)
>>>>>>>>>>>>
>>>>>>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이
>>>>>>>>>>>> 작성:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>>>>>>>> somewhere.
>>>>>>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS"
>>>>>>>>>>>>> when
>>>>>>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>>>>>>> Looks now it only leaves a link and does not change JIRA's
>>>>>>>>>>>>> status.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever
>>>>>>>>>>>>> since this happened, this makes (at least to me) duplicately check the
>>>>>>>>>>>>> JIRAs.
>>>>>>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Hyukjin Kwon <gu...@gmail.com>.
WDYT @Josh Rosen <jo...@databricks.com>?
Seems
https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L131-L142
this
isn't working anymore.
Does it make sense to move it to native Jira-GitHub integration
<https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>
?
It won't change JIRA status as we used to do but it might be better from a
cursory look. However, maybe I missed some context.
2020년 4월 30일 (목) 오전 2:46, Nicholas Chammas <ni...@gmail.com>님이
작성:
> Not sure what you mean. The native integration will auto-link from a Jira
> ticket to the PRs that mention that ticket. I don't think it will update
> the ticket's status, though.
>
> Would you like me to file a ticket with Infra and see what they say?
>
> On Tue, Apr 28, 2020 at 12:21 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Maybe it's time to switch. Do you know if we can still link the JIRA
>> against Github?
>> The script used to change the status of JIRA too but it stopped working
>> for a long time - I suspect this isn't a big deal.
>>
>> 2020년 4월 25일 (토) 오전 10:31, Nicholas Chammas <ni...@gmail.com>님이
>> 작성:
>>
>>> Have we asked Infra recently about enabling the native Jira-GitHub
>>> integration
>>> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>?
>>> Maybe we can deprecate the part of this script that updates Jira tickets
>>> with links to the PR and rely on the native integration instead. We use it
>>> at my day job, for example.
>>>
>>> On Fri, Apr 24, 2020 at 12:39 AM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Seems like this github_jira_sync.py
>>>> <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
>>>> seems stopped working completely now.
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-31532 <>
>>>> https://github.com/apache/spark/pull/28316
>>>> https://issues.apache.org/jira/browse/SPARK-31529 <>
>>>> https://github.com/apache/spark/pull/28315
>>>> https://issues.apache.org/jira/browse/SPARK-31528 <>
>>>> https://github.com/apache/spark/pull/28313
>>>>
>>>> Josh, would you mind taking a look please when you find some time?
>>>> There is a bunch of JIRAs now, and it is very confusing which JIRA is
>>>> in progress with a PR or not.
>>>>
>>>>
>>>> 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>
>>>>> Just FYI, I had to come up with a better JQL to filter out the JIRAs
>>>>> that already have linked PRs.
>>>>> In case it helps someone, I use this JQL now to look through the open
>>>>> JIRAs:
>>>>>
>>>>> project = SPARK AND
>>>>> status = Open AND
>>>>> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
>>>>> ORDER BY created DESC, priority DESC, updated DESC
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>
>>>>>> That's a great explanation. Thanks I didn't know that.
>>>>>>
>>>>>> Josh, do you know who I should ping on this?
>>>>>>
>>>>>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, Hyukjin.
>>>>>>>
>>>>>>> In short, there are two bots. And, the current situation happens
>>>>>>> when only one bot with `dev/github_jira_sync.py` works.
>>>>>>>
>>>>>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status
>>>>>>> change because it only use `add_remote_link` and `add_comment` API.
>>>>>>> I know only this bot (in Apache Spark repository repo)
>>>>>>>
>>>>>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>>>>>> `githubbot` (Name: `ASF GitHub Bot`).
>>>>>>> And, the other bot's activity is done under JIRA ID `apachespark`
>>>>>>> (Name: `Apache Spark`).
>>>>>>> The other bot is the one which Josh mentioned before. (in
>>>>>>> `databricks/spark-pr-dashboard` repo).
>>>>>>>
>>>>>>> The root cause will be the same. The API key used by the bot is
>>>>>>> rejected by Apache JIRA and forwarded to CAPCHAR.
>>>>>>>
>>>>>>> Bests,
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Seems this issue is re-happening again. Seems the PR link is
>>>>>>>> properly created in the corresponding JIRA but it doesn't change the JIRA's
>>>>>>>> status from OPEN to IN-PROGRESS.
>>>>>>>>
>>>>>>>> See, for instance,
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>>>>>
>>>>>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>>>>>
>>>>>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>>>
>>>>>>>>> Thank you so much Josh .. !!
>>>>>>>>>
>>>>>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>>>>>>>
>>>>>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>>>>>> )
>>>>>>>>>>
>>>>>>>>>> I checked the AppEngine logs and it looks like we're getting
>>>>>>>>>> error responses, possibly due to a credentials issue:
>>>>>>>>>>
>>>>>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>>>>>> Traceback (most recent call last): File
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>>>>>> line 138
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>>> issue_number)) File
>>>>>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>>> issue_number))
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>>> line 27
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>>>>>> jira_client = get_jira_client()
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>>> line 18
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>> line 472
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in __init__ si = self.server_info() File
>>>>>>>>>>> si = self.server_info()
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>> line 2133
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>>>>>> j = self._get_json('serverInfo')
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>>> line 2549
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>>>>>> r = self._session.get(url, params=params)
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>> line 151
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>> line 147
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>>> line 57
>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error,
>>>>>>>>>>> r.url, request=request, response=r, **kwargs)
>>>>>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so
>>>>>>>>>> I used the credentials to manually log in and complete the challenge.
>>>>>>>>>>
>>>>>>>>>> Hopefully that's enough to fix things, but to prevent
>>>>>>>>>> re-occurrence we might need to change the login credential type from
>>>>>>>>>> username + password to instead use OAuth tokens.
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <
>>>>>>>>>> gurwls223@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>>>>>>>> rapidly increased (from around 2400 to 2600)
>>>>>>>>>>>
>>>>>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이
>>>>>>>>>>> 작성:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>>>>>>> somewhere.
>>>>>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS"
>>>>>>>>>>>> when
>>>>>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>>>>>> Looks now it only leaves a link and does not change JIRA's
>>>>>>>>>>>> status.
>>>>>>>>>>>>
>>>>>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>>>>>
>>>>>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever since
>>>>>>>>>>>> this happened, this makes (at least to me) duplicately check the JIRAs.
>>>>>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Nicholas Chammas <ni...@gmail.com>.
Not sure what you mean. The native integration will auto-link from a Jira
ticket to the PRs that mention that ticket. I don't think it will update
the ticket's status, though.
Would you like me to file a ticket with Infra and see what they say?
On Tue, Apr 28, 2020 at 12:21 AM Hyukjin Kwon <gu...@gmail.com> wrote:
> Maybe it's time to switch. Do you know if we can still link the JIRA
> against Github?
> The script used to change the status of JIRA too but it stopped working
> for a long time - I suspect this isn't a big deal.
>
> 2020년 4월 25일 (토) 오전 10:31, Nicholas Chammas <ni...@gmail.com>님이
> 작성:
>
>> Have we asked Infra recently about enabling the native Jira-GitHub
>> integration
>> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>?
>> Maybe we can deprecate the part of this script that updates Jira tickets
>> with links to the PR and rely on the native integration instead. We use it
>> at my day job, for example.
>>
>> On Fri, Apr 24, 2020 at 12:39 AM Hyukjin Kwon <gu...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> Seems like this github_jira_sync.py
>>> <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
>>> seems stopped working completely now.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-31532 <>
>>> https://github.com/apache/spark/pull/28316
>>> https://issues.apache.org/jira/browse/SPARK-31529 <>
>>> https://github.com/apache/spark/pull/28315
>>> https://issues.apache.org/jira/browse/SPARK-31528 <>
>>> https://github.com/apache/spark/pull/28313
>>>
>>> Josh, would you mind taking a look please when you find some time?
>>> There is a bunch of JIRAs now, and it is very confusing which JIRA is in
>>> progress with a PR or not.
>>>
>>>
>>> 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>
>>>> Just FYI, I had to come up with a better JQL to filter out the JIRAs
>>>> that already have linked PRs.
>>>> In case it helps someone, I use this JQL now to look through the open
>>>> JIRAs:
>>>>
>>>> project = SPARK AND
>>>> status = Open AND
>>>> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
>>>> ORDER BY created DESC, priority DESC, updated DESC
>>>>
>>>>
>>>>
>>>>
>>>> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>
>>>>> That's a great explanation. Thanks I didn't know that.
>>>>>
>>>>> Josh, do you know who I should ping on this?
>>>>>
>>>>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, Hyukjin.
>>>>>>
>>>>>> In short, there are two bots. And, the current situation happens when
>>>>>> only one bot with `dev/github_jira_sync.py` works.
>>>>>>
>>>>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status
>>>>>> change because it only use `add_remote_link` and `add_comment` API.
>>>>>> I know only this bot (in Apache Spark repository repo)
>>>>>>
>>>>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>>>>> `githubbot` (Name: `ASF GitHub Bot`).
>>>>>> And, the other bot's activity is done under JIRA ID `apachespark`
>>>>>> (Name: `Apache Spark`).
>>>>>> The other bot is the one which Josh mentioned before. (in
>>>>>> `databricks/spark-pr-dashboard` repo).
>>>>>>
>>>>>> The root cause will be the same. The API key used by the bot is
>>>>>> rejected by Apache JIRA and forwarded to CAPCHAR.
>>>>>>
>>>>>> Bests,
>>>>>> Dongjoon.
>>>>>>
>>>>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Seems this issue is re-happening again. Seems the PR link is
>>>>>>> properly created in the corresponding JIRA but it doesn't change the JIRA's
>>>>>>> status from OPEN to IN-PROGRESS.
>>>>>>>
>>>>>>> See, for instance,
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>>>>
>>>>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>>>>
>>>>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>>
>>>>>>>> Thank you so much Josh .. !!
>>>>>>>>
>>>>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>>>>>>
>>>>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>> I checked the AppEngine logs and it looks like we're getting error
>>>>>>>>> responses, possibly due to a credentials issue:
>>>>>>>>>
>>>>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>>>>> Traceback (most recent call last): File
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>>>>> line 138
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>> issue_number)) File
>>>>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>>> issue_number))
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>> line 27
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>>>>> jira_client = get_jira_client()
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>>> line 18
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>> line 472
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in __init__ si = self.server_info() File
>>>>>>>>>> si = self.server_info()
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>> line 2133
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>>>>> j = self._get_json('serverInfo')
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>>> line 2549
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>>>>> r = self._session.get(url, params=params)
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>> line 151
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>> line 147
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>>> line 57
>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error,
>>>>>>>>>> r.url, request=request, response=r, **kwargs)
>>>>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so
>>>>>>>>> I used the credentials to manually log in and complete the challenge.
>>>>>>>>>
>>>>>>>>> Hopefully that's enough to fix things, but to prevent
>>>>>>>>> re-occurrence we might need to change the login credential type from
>>>>>>>>> username + password to instead use OAuth tokens.
>>>>>>>>>
>>>>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>>>>>>> rapidly increased (from around 2400 to 2600)
>>>>>>>>>>
>>>>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이
>>>>>>>>>> 작성:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>>>>>> somewhere.
>>>>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>>>>> Looks now it only leaves a link and does not change JIRA's
>>>>>>>>>>> status.
>>>>>>>>>>>
>>>>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>>>>
>>>>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever since
>>>>>>>>>>> this happened, this makes (at least to me) duplicately check the JIRAs.
>>>>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Hyukjin Kwon <gu...@gmail.com>.
Maybe it's time to switch. Do you know if we can still link the JIRA
against Github?
The script used to change the status of JIRA too but it stopped working for
a long time - I suspect this isn't a big deal.
2020년 4월 25일 (토) 오전 10:31, Nicholas Chammas <ni...@gmail.com>님이
작성:
> Have we asked Infra recently about enabling the native Jira-GitHub
> integration
> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>?
> Maybe we can deprecate the part of this script that updates Jira tickets
> with links to the PR and rely on the native integration instead. We use it
> at my day job, for example.
>
> On Fri, Apr 24, 2020 at 12:39 AM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Hi all,
>>
>> Seems like this github_jira_sync.py
>> <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
>> seems stopped working completely now.
>>
>> https://issues.apache.org/jira/browse/SPARK-31532 <>
>> https://github.com/apache/spark/pull/28316
>> https://issues.apache.org/jira/browse/SPARK-31529 <>
>> https://github.com/apache/spark/pull/28315
>> https://issues.apache.org/jira/browse/SPARK-31528 <>
>> https://github.com/apache/spark/pull/28313
>>
>> Josh, would you mind taking a look please when you find some time?
>> There is a bunch of JIRAs now, and it is very confusing which JIRA is in
>> progress with a PR or not.
>>
>>
>> 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>
>>> Just FYI, I had to come up with a better JQL to filter out the JIRAs
>>> that already have linked PRs.
>>> In case it helps someone, I use this JQL now to look through the open
>>> JIRAs:
>>>
>>> project = SPARK AND
>>> status = Open AND
>>> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
>>> ORDER BY created DESC, priority DESC, updated DESC
>>>
>>>
>>>
>>>
>>> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>
>>>> That's a great explanation. Thanks I didn't know that.
>>>>
>>>> Josh, do you know who I should ping on this?
>>>>
>>>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, Hyukjin.
>>>>>
>>>>> In short, there are two bots. And, the current situation happens when
>>>>> only one bot with `dev/github_jira_sync.py` works.
>>>>>
>>>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change
>>>>> because it only use `add_remote_link` and `add_comment` API.
>>>>> I know only this bot (in Apache Spark repository repo)
>>>>>
>>>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>>>> `githubbot` (Name: `ASF GitHub Bot`).
>>>>> And, the other bot's activity is done under JIRA ID `apachespark`
>>>>> (Name: `Apache Spark`).
>>>>> The other bot is the one which Josh mentioned before. (in
>>>>> `databricks/spark-pr-dashboard` repo).
>>>>>
>>>>> The root cause will be the same. The API key used by the bot is
>>>>> rejected by Apache JIRA and forwarded to CAPCHAR.
>>>>>
>>>>> Bests,
>>>>> Dongjoon.
>>>>>
>>>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Seems this issue is re-happening again. Seems the PR link is properly
>>>>>> created in the corresponding JIRA but it doesn't change the JIRA's status
>>>>>> from OPEN to IN-PROGRESS.
>>>>>>
>>>>>> See, for instance,
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>>>
>>>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>>>
>>>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>
>>>>>>> Thank you so much Josh .. !!
>>>>>>>
>>>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>>>>>
>>>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>>>> )
>>>>>>>>
>>>>>>>> I checked the AppEngine logs and it looks like we're getting error
>>>>>>>> responses, possibly due to a credentials issue:
>>>>>>>>
>>>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>>>> Traceback (most recent call last): File
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>>>> line 138
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>> issue_number)) File
>>>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>>> issue_number))
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>> line 27
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>>>> jira_client = get_jira_client()
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>>> line 18
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>> line 472
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in __init__ si = self.server_info() File
>>>>>>>>> si = self.server_info()
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>> line 2133
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>>>> j = self._get_json('serverInfo')
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>>> line 2549
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>>>> r = self._session.get(url, params=params)
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>> line 151
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>> line 147
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>>> line 57
>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error,
>>>>>>>>> r.url, request=request, response=r, **kwargs)
>>>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>>>
>>>>>>>>
>>>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so I
>>>>>>>> used the credentials to manually log in and complete the challenge.
>>>>>>>>
>>>>>>>> Hopefully that's enough to fix things, but to prevent re-occurrence
>>>>>>>> we might need to change the login credential type from username + password
>>>>>>>> to instead use OAuth tokens.
>>>>>>>>
>>>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>>>>>> rapidly increased (from around 2400 to 2600)
>>>>>>>>>
>>>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>>>>> somewhere.
>>>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>>>>>>>
>>>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>>>
>>>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever since
>>>>>>>>>> this happened, this makes (at least to me) duplicately check the JIRAs.
>>>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Nicholas Chammas <ni...@gmail.com>.
Have we asked Infra recently about enabling the native Jira-GitHub
integration
<https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>?
Maybe we can deprecate the part of this script that updates Jira tickets
with links to the PR and rely on the native integration instead. We use it
at my day job, for example.
On Fri, Apr 24, 2020 at 12:39 AM Hyukjin Kwon <gu...@gmail.com> wrote:
> Hi all,
>
> Seems like this github_jira_sync.py
> <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
> seems stopped working completely now.
>
> https://issues.apache.org/jira/browse/SPARK-31532 <>
> https://github.com/apache/spark/pull/28316
> https://issues.apache.org/jira/browse/SPARK-31529 <>
> https://github.com/apache/spark/pull/28315
> https://issues.apache.org/jira/browse/SPARK-31528 <>
> https://github.com/apache/spark/pull/28313
>
> Josh, would you mind taking a look please when you find some time?
> There is a bunch of JIRAs now, and it is very confusing which JIRA is in
> progress with a PR or not.
>
>
> 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> Just FYI, I had to come up with a better JQL to filter out the JIRAs that
>> already have linked PRs.
>> In case it helps someone, I use this JQL now to look through the open
>> JIRAs:
>>
>> project = SPARK AND
>> status = Open AND
>> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
>> ORDER BY created DESC, priority DESC, updated DESC
>>
>>
>>
>>
>> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>
>>> That's a great explanation. Thanks I didn't know that.
>>>
>>> Josh, do you know who I should ping on this?
>>>
>>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com>
>>> wrote:
>>>
>>>> Hi, Hyukjin.
>>>>
>>>> In short, there are two bots. And, the current situation happens when
>>>> only one bot with `dev/github_jira_sync.py` works.
>>>>
>>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change
>>>> because it only use `add_remote_link` and `add_comment` API.
>>>> I know only this bot (in Apache Spark repository repo)
>>>>
>>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>>> `githubbot` (Name: `ASF GitHub Bot`).
>>>> And, the other bot's activity is done under JIRA ID `apachespark`
>>>> (Name: `Apache Spark`).
>>>> The other bot is the one which Josh mentioned before. (in
>>>> `databricks/spark-pr-dashboard` repo).
>>>>
>>>> The root cause will be the same. The API key used by the bot is
>>>> rejected by Apache JIRA and forwarded to CAPCHAR.
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Seems this issue is re-happening again. Seems the PR link is properly
>>>>> created in the corresponding JIRA but it doesn't change the JIRA's status
>>>>> from OPEN to IN-PROGRESS.
>>>>>
>>>>> See, for instance,
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>>
>>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>>
>>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>
>>>>>> Thank you so much Josh .. !!
>>>>>>
>>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>>>>
>>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>>> )
>>>>>>>
>>>>>>> I checked the AppEngine logs and it looks like we're getting error
>>>>>>> responses, possibly due to a credentials issue:
>>>>>>>
>>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>>> Traceback (most recent call last): File
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>>> line 138
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>> issue_number)) File
>>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>>> issue_number))
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>> line 27
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>>> jira_client = get_jira_client()
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>>> line 18
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>> line 472
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in __init__ si = self.server_info() File
>>>>>>>> si = self.server_info()
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>> line 2133
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>>> j = self._get_json('serverInfo')
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>>> line 2549
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>>> r = self._session.get(url, params=params)
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>> line 151
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>> line 147
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>>> line 57
>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error,
>>>>>>>> r.url, request=request, response=r, **kwargs)
>>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>>
>>>>>>>
>>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so I
>>>>>>> used the credentials to manually log in and complete the challenge.
>>>>>>>
>>>>>>> Hopefully that's enough to fix things, but to prevent re-occurrence
>>>>>>> we might need to change the login credential type from username + password
>>>>>>> to instead use OAuth tokens.
>>>>>>>
>>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>>>>> rapidly increased (from around 2400 to 2600)
>>>>>>>>
>>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>>>> somewhere.
>>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>>>>>>
>>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>>
>>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever since
>>>>>>>>> this happened, this makes (at least to me) duplicately check the JIRAs.
>>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Hyukjin Kwon <gu...@gmail.com>.
Hi all,
Seems like this github_jira_sync.py
<https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
seems stopped working completely now.
https://issues.apache.org/jira/browse/SPARK-31532 <>
https://github.com/apache/spark/pull/28316
https://issues.apache.org/jira/browse/SPARK-31529 <>
https://github.com/apache/spark/pull/28315
https://issues.apache.org/jira/browse/SPARK-31528 <>
https://github.com/apache/spark/pull/28313
Josh, would you mind taking a look please when you find some time?
There is a bunch of JIRAs now, and it is very confusing which JIRA is in
progress with a PR or not.
2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> Just FYI, I had to come up with a better JQL to filter out the JIRAs that
> already have linked PRs.
> In case it helps someone, I use this JQL now to look through the open
> JIRAs:
>
> project = SPARK AND
> status = Open AND
> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
> ORDER BY created DESC, priority DESC, updated DESC
>
>
>
>
> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> That's a great explanation. Thanks I didn't know that.
>>
>> Josh, do you know who I should ping on this?
>>
>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com>
>> wrote:
>>
>>> Hi, Hyukjin.
>>>
>>> In short, there are two bots. And, the current situation happens when
>>> only one bot with `dev/github_jira_sync.py` works.
>>>
>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change
>>> because it only use `add_remote_link` and `add_comment` API.
>>> I know only this bot (in Apache Spark repository repo)
>>>
>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>> `githubbot` (Name: `ASF GitHub Bot`).
>>> And, the other bot's activity is done under JIRA ID `apachespark` (Name:
>>> `Apache Spark`).
>>> The other bot is the one which Josh mentioned before. (in
>>> `databricks/spark-pr-dashboard` repo).
>>>
>>> The root cause will be the same. The API key used by the bot is rejected
>>> by Apache JIRA and forwarded to CAPCHAR.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Seems this issue is re-happening again. Seems the PR link is properly
>>>> created in the corresponding JIRA but it doesn't change the JIRA's status
>>>> from OPEN to IN-PROGRESS.
>>>>
>>>> See, for instance,
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>
>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>
>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>
>>>>> Thank you so much Josh .. !!
>>>>>
>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>>>
>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>> )
>>>>>>
>>>>>> I checked the AppEngine logs and it looks like we're getting error
>>>>>> responses, possibly due to a credentials issue:
>>>>>>
>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>> Traceback (most recent call last): File
>>>>>>> Traceback (most recent call last):
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>> line 138
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>> issue_number)) File
>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>> issue_number))
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>> line 27
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>> jira_client = get_jira_client()
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>> line 18
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>> line 472
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in __init__ si = self.server_info() File
>>>>>>> si = self.server_info()
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>> line 2133
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>> j = self._get_json('serverInfo')
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>> line 2549
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>> r = self._session.get(url, params=params)
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>> line 151
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>> line 147
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>> line 57
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error,
>>>>>>> r.url, request=request, response=r, **kwargs)
>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>
>>>>>>
>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so I
>>>>>> used the credentials to manually log in and complete the challenge.
>>>>>>
>>>>>> Hopefully that's enough to fix things, but to prevent re-occurrence
>>>>>> we might need to change the login credential type from username + password
>>>>>> to instead use OAuth tokens.
>>>>>>
>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>>>> rapidly increased (from around 2400 to 2600)
>>>>>>>
>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>>> somewhere.
>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>>>>>
>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>
>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever since
>>>>>>>> this happened, this makes (at least to me) duplicately check the JIRAs.
>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Hyukjin Kwon <gu...@gmail.com>.
Just FYI, I had to come up with a better JQL to filter out the JIRAs that
already have linked PRs.
In case it helps someone, I use this JQL now to look through the open JIRAs:
project = SPARK AND
status = Open AND
NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
ORDER BY created DESC, priority DESC, updated DESC
2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> That's a great explanation. Thanks I didn't know that.
>
> Josh, do you know who I should ping on this?
>
> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com> wrote:
>
>> Hi, Hyukjin.
>>
>> In short, there are two bots. And, the current situation happens when
>> only one bot with `dev/github_jira_sync.py` works.
>>
>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change
>> because it only use `add_remote_link` and `add_comment` API.
>> I know only this bot (in Apache Spark repository repo)
>>
>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>> `githubbot` (Name: `ASF GitHub Bot`).
>> And, the other bot's activity is done under JIRA ID `apachespark` (Name:
>> `Apache Spark`).
>> The other bot is the one which Josh mentioned before. (in
>> `databricks/spark-pr-dashboard` repo).
>>
>> The root cause will be the same. The API key used by the bot is rejected
>> by Apache JIRA and forwarded to CAPCHAR.
>>
>> Bests,
>> Dongjoon.
>>
>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> Seems this issue is re-happening again. Seems the PR link is properly
>>> created in the corresponding JIRA but it doesn't change the JIRA's status
>>> from OPEN to IN-PROGRESS.
>>>
>>> See, for instance,
>>>
>>> https://issues.apache.org/jira/browse/SPARK-28443
>>> https://issues.apache.org/jira/browse/SPARK-28440
>>> https://issues.apache.org/jira/browse/SPARK-28436
>>> https://issues.apache.org/jira/browse/SPARK-28434
>>> https://issues.apache.org/jira/browse/SPARK-28433
>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>
>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>
>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>
>>>> Thank you so much Josh .. !!
>>>>
>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>>
>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>> )
>>>>>
>>>>> I checked the AppEngine logs and it looks like we're getting error
>>>>> responses, possibly due to a credentials issue:
>>>>>
>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>> Traceback (most recent call last): File
>>>>>> Traceback (most recent call last):
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>> line 138
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>> issue_number)) File
>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>> issue_number))
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>> line 27
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>> jira_client = get_jira_client()
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>> line 18
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>> app.config['JIRA_PASSWORD']))
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>> line 472
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in __init__ si = self.server_info() File
>>>>>> si = self.server_info()
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>> line 2133
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>> j = self._get_json('serverInfo')
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>> line 2549
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>> r = self._session.get(url, params=params)
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>> line 151
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>> line 147
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>> line 57
>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>> CAPTCHA_CHALLENGE; login-url=https://issues.apache.org/jira/login.jsp r.status_code,
>>>>>> error, r.url, request=request, response=r, **kwargs)
>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>
>>>>>
>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so I
>>>>> used the credentials to manually log in and complete the challenge.
>>>>>
>>>>> Hopefully that's enough to fix things, but to prevent re-occurrence we
>>>>> might need to change the login credential type from username + password to
>>>>> instead use OAuth tokens.
>>>>>
>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>>> rapidly increased (from around 2400 to 2600)
>>>>>>
>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>> somewhere.
>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>>>>> somebody opens a PR against a JIRA.
>>>>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>>>>
>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>
>>>>>>> FWIW, I check every PR and JIRA almost every day but ever since this
>>>>>>> happened, this makes (at least to me) duplicately check the JIRAs.
>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Hyukjin Kwon <gu...@gmail.com>.
That's a great explanation. Thanks I didn't know that.
Josh, do you know who I should ping on this?
On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <do...@gmail.com> wrote:
> Hi, Hyukjin.
>
> In short, there are two bots. And, the current situation happens when only
> one bot with `dev/github_jira_sync.py` works.
>
> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change
> because it only use `add_remote_link` and `add_comment` API.
> I know only this bot (in Apache Spark repository repo)
>
> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
> `githubbot` (Name: `ASF GitHub Bot`).
> And, the other bot's activity is done under JIRA ID `apachespark` (Name:
> `Apache Spark`).
> The other bot is the one which Josh mentioned before. (in
> `databricks/spark-pr-dashboard` repo).
>
> The root cause will be the same. The API key used by the bot is rejected
> by Apache JIRA and forwarded to CAPCHAR.
>
> Bests,
> Dongjoon.
>
> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Hi all,
>>
>> Seems this issue is re-happening again. Seems the PR link is properly
>> created in the corresponding JIRA but it doesn't change the JIRA's status
>> from OPEN to IN-PROGRESS.
>>
>> See, for instance,
>>
>> https://issues.apache.org/jira/browse/SPARK-28443
>> https://issues.apache.org/jira/browse/SPARK-28440
>> https://issues.apache.org/jira/browse/SPARK-28436
>> https://issues.apache.org/jira/browse/SPARK-28434
>> https://issues.apache.org/jira/browse/SPARK-28433
>> https://issues.apache.org/jira/browse/SPARK-28431
>>
>> Josh and Dongjoon, do you guys maybe have any idea?
>>
>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>
>>> Thank you so much Josh .. !!
>>>
>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>>
>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>> )
>>>>
>>>> I checked the AppEngine logs and it looks like we're getting error
>>>> responses, possibly due to a credentials issue:
>>>>
>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>> Traceback (most recent call last): File
>>>>> Traceback (most recent call last):
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>> line 138
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>> issue_number)) File
>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>> issue_number))
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>> line 27
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>> jira_client = get_jira_client()
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>> line 18
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>> app.config['JIRA_PASSWORD']))
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>> line 472
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in __init__ si = self.server_info() File
>>>>> si = self.server_info()
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>> line 2133
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in server_info j = self._get_json('serverInfo') File
>>>>> j = self._get_json('serverInfo')
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>> line 2549
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>> r = self._session.get(url, params=params)
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>> line 151
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>> return self.__verb('GET', url, **kwargs)
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>> line 147
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>> line 57
>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>> CAPTCHA_CHALLENGE; login-url=https://issues.apache.org/jira/login.jsp r.status_code,
>>>>> error, r.url, request=request, response=r, **kwargs)
>>>>> JIRAError: JiraError HTTP 403 url:
>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>> https://issues.apache.org/jira/login.jsp
>>>>
>>>>
>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so I
>>>> used the credentials to manually log in and complete the challenge.
>>>>
>>>> Hopefully that's enough to fix things, but to prevent re-occurrence we
>>>> might need to change the login credential type from username + password to
>>>> instead use OAuth tokens.
>>>>
>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>>>> wrote:
>>>>
>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>> rapidly increased (from around 2400 to 2600)
>>>>>
>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>> somewhere.
>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>>>> somebody opens a PR against a JIRA.
>>>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>>>
>>>>>> Can someone else who knows where it's running can check this?
>>>>>>
>>>>>> FWIW, I check every PR and JIRA almost every day but ever since this
>>>>>> happened, this makes (at least to me) duplicately check the JIRAs.
>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>> duplicated because JIRAs having PRs have different status, "IN PROGRESS"
>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>
Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Posted by Dongjoon Hyun <do...@gmail.com>.
Hi, Hyukjin.
In short, there are two bots. And, the current situation happens when only
one bot with `dev/github_jira_sync.py` works.
And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change
because it only use `add_remote_link` and `add_comment` API.
I know only this bot (in Apache Spark repository repo)
AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
`githubbot` (Name: `ASF GitHub Bot`).
And, the other bot's activity is done under JIRA ID `apachespark` (Name:
`Apache Spark`).
The other bot is the one which Josh mentioned before. (in
`databricks/spark-pr-dashboard` repo).
The root cause will be the same. The API key used by the bot is rejected by
Apache JIRA and forwarded to CAPCHAR.
Bests,
Dongjoon.
On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gu...@gmail.com> wrote:
> Hi all,
>
> Seems this issue is re-happening again. Seems the PR link is properly
> created in the corresponding JIRA but it doesn't change the JIRA's status
> from OPEN to IN-PROGRESS.
>
> See, for instance,
>
> https://issues.apache.org/jira/browse/SPARK-28443
> https://issues.apache.org/jira/browse/SPARK-28440
> https://issues.apache.org/jira/browse/SPARK-28436
> https://issues.apache.org/jira/browse/SPARK-28434
> https://issues.apache.org/jira/browse/SPARK-28433
> https://issues.apache.org/jira/browse/SPARK-28431
>
> Josh and Dongjoon, do you guys maybe have any idea?
>
> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> Thank you so much Josh .. !!
>>
>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <ro...@gmail.com>님이 작성:
>>
>>> The code for this runs in http://spark-prs.appspot.com (see
>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>> )
>>>
>>> I checked the AppEngine logs and it looks like we're getting error
>>> responses, possibly due to a credentials issue:
>>>
>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>> Traceback (most recent call last): File
>>>> Traceback (most recent call last):
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>> line 138
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>> issue_number)) File
>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>> issue_number))
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>> line 27
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in start_issue_progress jira_client = get_jira_client() File
>>>> jira_client = get_jira_client()
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>> line 18
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>> app.config['JIRA_PASSWORD']))
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>> line 472
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in __init__ si = self.server_info() File
>>>> si = self.server_info()
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>> line 2133
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in server_info j = self._get_json('serverInfo') File
>>>> j = self._get_json('serverInfo')
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>> line 2549
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in _get_json r = self._session.get(url, params=params) File
>>>> r = self._session.get(url, params=params)
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>> line 151
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in get return self.__verb('GET', url, **kwargs) File
>>>> return self.__verb('GET', url, **kwargs)
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>> line 147
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>> raise_on_error(response, verb=verb, **kwargs)
>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>> line 57
>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>> CAPTCHA_CHALLENGE; login-url=https://issues.apache.org/jira/login.jsp r.status_code,
>>>> error, r.url, request=request, response=r, **kwargs)
>>>> JIRAError: JiraError HTTP 403 url:
>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>> https://issues.apache.org/jira/login.jsp
>>>
>>>
>>> It looks like ASF JIRA was throwing a captcha challenge at us, so I used
>>> the credentials to manually log in and complete the challenge.
>>>
>>> Hopefully that's enough to fix things, but to prevent re-occurrence we
>>> might need to change the login credential type from username + password to
>>> instead use OAuth tokens.
>>>
>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>> rapidly increased (from around 2400 to 2600)
>>>>
>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>> somewhere.
>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>>> somebody opens a PR against a JIRA.
>>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>>
>>>>> Can someone else who knows where it's running can check this?
>>>>>
>>>>> FWIW, I check every PR and JIRA almost every day but ever since this
>>>>> happened, this makes (at least to me) duplicately check the JIRAs.
>>>>> Previously, if I check all the PRs and JIRAs, they were not duplicated
>>>>> because JIRAs having PRs have different status, "IN PROGRESS" but now all
>>>>> JIRAs have "OPEN" status.
>>>>>
>>>>> Thanks.
>>>>>
>>>>