You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Bryan Cutler <cu...@gmail.com> on 2017/08/02 18:46:49 UTC
Some PRs not automatically linked to JIRAs
Hi Devs,
I've noticed a couple PRs recently have not been automatically linked to
the related JIRAs. This was one of mine (I linked it manually)
https://issues.apache.org/jira/browse/SPARK-21583, but I've seen it happen
elsewhere. I think this is the script that does it, but it hasn't been
changed recently
https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
Anyone else seen this or know what's going on?
Thanks,
Bryan
Re: Some PRs not automatically linked to JIRAs
Posted by Hyukjin Kwon <gu...@gmail.com>.
This issue is still persistent.
https://issues.apache.org/jira/browse/SPARK-26132
https://issues.apache.org/jira/browse/SPARK-26129
https://issues.apache.org/jira/browse/SPARK-26127
https://issues.apache.org/jira/browse/SPARK-26109
https://issues.apache.org/jira/browse/SPARK-26106
https://issues.apache.org/jira/browse/SPARK-26104
the links are still duplicated. Looks the scripts are being ran at multiple
places.
2018년 10월 30일 (화) 오후 4:58, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> Duplicated link problem looks still persistent:
>
> https://issues.apache.org/jira/browse/SPARK-25881
> https://issues.apache.org/jira/browse/SPARK-25880
>
> I suspect if there are two places that runs this script. Not a big deal
> but people that can fix this are specific.
> I am leaving another reminder here.
>
> Thanks.
>
>
> 2018년 10월 1일 (월) 오후 7:15, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> Seems fixed but looks it starts to leave duplicated PR links for some
>> recent JIRAs. Not a big deal but are they being ran in multiple places
>> maybe?
>>
>> For instance,
>>
>> https://issues.apache.org/jira/browse/SPARK-25579
>> https://issues.apache.org/jira/browse/SPARK-25574
>> https://issues.apache.org/jira/browse/SPARK-25564
>>
>>
>> 2018년 9월 17일 (월) 오후 10:09, Ilan Filonenko <il...@gmail.com>님이 작성:
>>
>>> Same over here:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-25291 /
>>> https://github.com/apache/spark/pull/22415
>>>
>>> On Sun, Sep 16, 2018 at 10:09 PM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> Seems same thing is happening again.
>>>>
>>>> For instance,
>>>> - https://issues.apache.org/jira/browse/SPARK-25440 /
>>>> https://github.com/apache/spark/pull/22429
>>>> - https://issues.apache.org/jira/browse/SPARK-25429 /
>>>> https://github.com/apache/spark/pull/22420
>>>>
>>>>
>>>> 2017년 8월 3일 (목) 오전 9:06, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>
>>>>> I think it works for anyone who can leave a web link and comment.
>>>>>
>>>>>
>>>>> For "in progress" Resolution, it looks I am unable to manually set.
>>>>> Please let me know if anyone knows.
>>>>>
>>>>>
>>>>> For single JIRA, I manually modified the script to process single item
>>>>> before. I guess you know what the script does but want to rather test,
>>>>> right?
>>>>>
>>>>> This looks requiring Python JIRA client, which I installed by pip
>>>>> install jira==1.0.7 and I personally tested (and just double checked this)
>>>>> after commenting out JIRA related codes:
>>>>>
>>>>>
>>>>> diff --git a/dev/github_jira_sync.py b/dev/github_jira_sync.py
>>>>> index acc9aeabbb9..c8adf0c4ade 100755
>>>>> --- a/dev/github_jira_sync.py
>>>>> +++ b/dev/github_jira_sync.py
>>>>> @@ -101,8 +101,8 @@ def get_max_pr():
>>>>> return 0
>>>>>
>>>>>
>>>>> -jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>>>> - basic_auth=(JIRA_USERNAME,
>>>>> JIRA_PASSWORD))
>>>>> +# jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>>>> +# basic_auth=(JIRA_USERNAME,
>>>>> JIRA_PASSWORD))
>>>>>
>>>>> jira_prs = get_jira_prs()
>>>>>
>>>>> @@ -124,14 +124,15 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>>>> int(kv[1]['number'])):
>>>>> url = pr['html_url']
>>>>> title = "[Github] Pull Request #%s (%s)" % (pr['number'],
>>>>> pr['user']['login'])
>>>>> try:
>>>>> - existing_links = map(lambda l: l.raw['object']['url'],
>>>>> jira_client.remote_links(issue))
>>>>> + # existing_links = map(lambda l: l.raw['object']['url'],
>>>>> jira_client.remote_links(issue))
>>>>> + pass
>>>>> except:
>>>>> print("Failure reading JIRA %s (does it exist?)" % issue)
>>>>> print(sys.exc_info()[0])
>>>>> continue
>>>>>
>>>>> - if url in existing_links:
>>>>> - continue
>>>>> + # if url in existing_links:
>>>>> + # continue
>>>>>
>>>>> icon = {"title": "Pull request #%s" % pr['number'],
>>>>> "url16x16": "https://assets-cdn.github.com/favicon.ico"}
>>>>> @@ -139,12 +140,12 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>>>> int(kv[1]['number'])):
>>>>> # For all possible fields see:
>>>>> #
>>>>> https://developer.atlassian.com/display/JIRADEV/Fields+in+Remote+Issue+Links
>>>>> # application = {"name": "Github pull requests", "type":
>>>>> "org.apache.spark.jira.github"}
>>>>> - jira_client.add_remote_link(issue, destination)
>>>>> + # jira_client.add_remote_link(issue, destination)
>>>>>
>>>>> comment = "User '%s' has created a pull request for this issue:"
>>>>> % pr['user']['login']
>>>>> comment += "\n%s" % pr['html_url']
>>>>> - if pr_num >= MIN_COMMENT_PR:
>>>>> - jira_client.add_comment(issue, comment)
>>>>> + # if pr_num >= MIN_COMMENT_PR:
>>>>> + # jira_client.add_comment(issue, comment)
>>>>>
>>>>> print("Added link %s <-> PR #%s" % (issue, pr['number']))
>>>>> num_updates += 1
>>>>>
>>>>>
>>>>> and
>>>>>
>>>>>
>>>>> rm .github-jira-max
>>>>> ./dev/github_jira_sync.py
>>>>>
>>>>>
>>>>>
>>>>> 2017-08-03 5:20 GMT+09:00 Josh Rosen <jo...@databricks.com>:
>>>>>
>>>>>> Usually the backend of https://spark-prs.appspot.com does the
>>>>>> linking while processing PR update tasks. It appears that the site's
>>>>>> connections to JIRA have started failing:
>>>>>>
>>>>>> ConnectionError: ('Connection aborted.', HTTPException('Deadline
>>>>>> exceeded while waiting for HTTP response from URL:
>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo',))
>>>>>>
>>>>>> From Stackdriver's log-based metrics, I can spot that this problem
>>>>>> started around July 24th. We're already using a much-higher-than-default
>>>>>> URL fetch timeout, so it's possible that the problem is related to access
>>>>>> credentials, IP blocks, outdated client libraries, or something else.
>>>>>>
>>>>>> On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Hyukjin! I didn't see your previous message.. It looks like
>>>>>>> your manual run worked pretty well for the JIRAs I'm following, the only
>>>>>>> thing is that it didn't mark them as "in progress", but that's not a big
>>>>>>> deal. Otherwise that helps until we can find out why it's not doing this
>>>>>>> automatically. I'm not familiar with that script, can anyone run it to
>>>>>>> apply to a single JIRA they are working on?
>>>>>>>
>>>>>>> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I was wondering about this too..
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, actually, I have been manually adding some links by resembling
>>>>>>>> the same steps in the script before.
>>>>>>>>
>>>>>>>> I was thinking it'd rather be nicer to run this manually once and
>>>>>>>> then I ran this against single JIRA
>>>>>>>>
>>>>>>>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show
>>>>>>>> how it looks like and check if there
>>>>>>>>
>>>>>>>> is any issue or objection just in case.
>>>>>>>>
>>>>>>>>
>>>>>>>> Will run this manually now once. I will revert all my action
>>>>>>>> manually if there is any issue by doing this.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>>>>>>>>
>>>>>>>>> Hyukjin mentioned this here earlier today and had run it manually,
>>>>>>>>> but yeah I'm not sure where it normally runs or why it hasn't. Shane not
>>>>>>>>> sure if you're the person to ask?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Devs,
>>>>>>>>>>
>>>>>>>>>> I've noticed a couple PRs recently have not been automatically
>>>>>>>>>> linked to the related JIRAs. This was one of mine (I linked it manually)
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen
>>>>>>>>>> it happen elsewhere. I think this is the script that does it, but it
>>>>>>>>>> hasn't been changed recently
>>>>>>>>>> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
>>>>>>>>>> Anyone else seen this or know what's going on?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bryan
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
Re: Some PRs not automatically linked to JIRAs
Posted by Hyukjin Kwon <gu...@gmail.com>.
Duplicated link problem looks still persistent:
https://issues.apache.org/jira/browse/SPARK-25881
https://issues.apache.org/jira/browse/SPARK-25880
I suspect if there are two places that runs this script. Not a big deal but
people that can fix this are specific.
I am leaving another reminder here.
Thanks.
2018년 10월 1일 (월) 오후 7:15, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> Seems fixed but looks it starts to leave duplicated PR links for some
> recent JIRAs. Not a big deal but are they being ran in multiple places
> maybe?
>
> For instance,
>
> https://issues.apache.org/jira/browse/SPARK-25579
> https://issues.apache.org/jira/browse/SPARK-25574
> https://issues.apache.org/jira/browse/SPARK-25564
>
>
> 2018년 9월 17일 (월) 오후 10:09, Ilan Filonenko <il...@gmail.com>님이 작성:
>
>> Same over here:
>>
>> https://issues.apache.org/jira/browse/SPARK-25291 /
>> https://github.com/apache/spark/pull/22415
>>
>> On Sun, Sep 16, 2018 at 10:09 PM Hyukjin Kwon <gu...@gmail.com>
>> wrote:
>>
>>> Seems same thing is happening again.
>>>
>>> For instance,
>>> - https://issues.apache.org/jira/browse/SPARK-25440 /
>>> https://github.com/apache/spark/pull/22429
>>> - https://issues.apache.org/jira/browse/SPARK-25429 /
>>> https://github.com/apache/spark/pull/22420
>>>
>>>
>>> 2017년 8월 3일 (목) 오전 9:06, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>
>>>> I think it works for anyone who can leave a web link and comment.
>>>>
>>>>
>>>> For "in progress" Resolution, it looks I am unable to manually set.
>>>> Please let me know if anyone knows.
>>>>
>>>>
>>>> For single JIRA, I manually modified the script to process single item
>>>> before. I guess you know what the script does but want to rather test,
>>>> right?
>>>>
>>>> This looks requiring Python JIRA client, which I installed by pip
>>>> install jira==1.0.7 and I personally tested (and just double checked this)
>>>> after commenting out JIRA related codes:
>>>>
>>>>
>>>> diff --git a/dev/github_jira_sync.py b/dev/github_jira_sync.py
>>>> index acc9aeabbb9..c8adf0c4ade 100755
>>>> --- a/dev/github_jira_sync.py
>>>> +++ b/dev/github_jira_sync.py
>>>> @@ -101,8 +101,8 @@ def get_max_pr():
>>>> return 0
>>>>
>>>>
>>>> -jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>>> - basic_auth=(JIRA_USERNAME,
>>>> JIRA_PASSWORD))
>>>> +# jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>>> +# basic_auth=(JIRA_USERNAME,
>>>> JIRA_PASSWORD))
>>>>
>>>> jira_prs = get_jira_prs()
>>>>
>>>> @@ -124,14 +124,15 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>>> int(kv[1]['number'])):
>>>> url = pr['html_url']
>>>> title = "[Github] Pull Request #%s (%s)" % (pr['number'],
>>>> pr['user']['login'])
>>>> try:
>>>> - existing_links = map(lambda l: l.raw['object']['url'],
>>>> jira_client.remote_links(issue))
>>>> + # existing_links = map(lambda l: l.raw['object']['url'],
>>>> jira_client.remote_links(issue))
>>>> + pass
>>>> except:
>>>> print("Failure reading JIRA %s (does it exist?)" % issue)
>>>> print(sys.exc_info()[0])
>>>> continue
>>>>
>>>> - if url in existing_links:
>>>> - continue
>>>> + # if url in existing_links:
>>>> + # continue
>>>>
>>>> icon = {"title": "Pull request #%s" % pr['number'],
>>>> "url16x16": "https://assets-cdn.github.com/favicon.ico"}
>>>> @@ -139,12 +140,12 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>>> int(kv[1]['number'])):
>>>> # For all possible fields see:
>>>> #
>>>> https://developer.atlassian.com/display/JIRADEV/Fields+in+Remote+Issue+Links
>>>> # application = {"name": "Github pull requests", "type":
>>>> "org.apache.spark.jira.github"}
>>>> - jira_client.add_remote_link(issue, destination)
>>>> + # jira_client.add_remote_link(issue, destination)
>>>>
>>>> comment = "User '%s' has created a pull request for this issue:" %
>>>> pr['user']['login']
>>>> comment += "\n%s" % pr['html_url']
>>>> - if pr_num >= MIN_COMMENT_PR:
>>>> - jira_client.add_comment(issue, comment)
>>>> + # if pr_num >= MIN_COMMENT_PR:
>>>> + # jira_client.add_comment(issue, comment)
>>>>
>>>> print("Added link %s <-> PR #%s" % (issue, pr['number']))
>>>> num_updates += 1
>>>>
>>>>
>>>> and
>>>>
>>>>
>>>> rm .github-jira-max
>>>> ./dev/github_jira_sync.py
>>>>
>>>>
>>>>
>>>> 2017-08-03 5:20 GMT+09:00 Josh Rosen <jo...@databricks.com>:
>>>>
>>>>> Usually the backend of https://spark-prs.appspot.com does the linking
>>>>> while processing PR update tasks. It appears that the site's connections to
>>>>> JIRA have started failing:
>>>>>
>>>>> ConnectionError: ('Connection aborted.', HTTPException('Deadline
>>>>> exceeded while waiting for HTTP response from URL:
>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo',))
>>>>>
>>>>> From Stackdriver's log-based metrics, I can spot that this problem
>>>>> started around July 24th. We're already using a much-higher-than-default
>>>>> URL fetch timeout, so it's possible that the problem is related to access
>>>>> credentials, IP blocks, outdated client libraries, or something else.
>>>>>
>>>>> On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cu...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Hyukjin! I didn't see your previous message.. It looks like
>>>>>> your manual run worked pretty well for the JIRAs I'm following, the only
>>>>>> thing is that it didn't mark them as "in progress", but that's not a big
>>>>>> deal. Otherwise that helps until we can find out why it's not doing this
>>>>>> automatically. I'm not familiar with that script, can anyone run it to
>>>>>> apply to a single JIRA they are working on?
>>>>>>
>>>>>> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I was wondering about this too..
>>>>>>>
>>>>>>>
>>>>>>> Yes, actually, I have been manually adding some links by resembling
>>>>>>> the same steps in the script before.
>>>>>>>
>>>>>>> I was thinking it'd rather be nicer to run this manually once and
>>>>>>> then I ran this against single JIRA
>>>>>>>
>>>>>>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show
>>>>>>> how it looks like and check if there
>>>>>>>
>>>>>>> is any issue or objection just in case.
>>>>>>>
>>>>>>>
>>>>>>> Will run this manually now once. I will revert all my action
>>>>>>> manually if there is any issue by doing this.
>>>>>>>
>>>>>>>
>>>>>>> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>>>>>>>
>>>>>>>> Hyukjin mentioned this here earlier today and had run it manually,
>>>>>>>> but yeah I'm not sure where it normally runs or why it hasn't. Shane not
>>>>>>>> sure if you're the person to ask?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Devs,
>>>>>>>>>
>>>>>>>>> I've noticed a couple PRs recently have not been automatically
>>>>>>>>> linked to the related JIRAs. This was one of mine (I linked it manually)
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen
>>>>>>>>> it happen elsewhere. I think this is the script that does it, but it
>>>>>>>>> hasn't been changed recently
>>>>>>>>> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
>>>>>>>>> Anyone else seen this or know what's going on?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bryan
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
Re: Some PRs not automatically linked to JIRAs
Posted by Hyukjin Kwon <gu...@gmail.com>.
Seems fixed but looks it starts to leave duplicated PR links for some
recent JIRAs. Not a big deal but are they being ran in multiple places
maybe?
For instance,
https://issues.apache.org/jira/browse/SPARK-25579
https://issues.apache.org/jira/browse/SPARK-25574
https://issues.apache.org/jira/browse/SPARK-25564
2018년 9월 17일 (월) 오후 10:09, Ilan Filonenko <il...@gmail.com>님이 작성:
> Same over here:
>
> https://issues.apache.org/jira/browse/SPARK-25291 /
> https://github.com/apache/spark/pull/22415
>
> On Sun, Sep 16, 2018 at 10:09 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Seems same thing is happening again.
>>
>> For instance,
>> - https://issues.apache.org/jira/browse/SPARK-25440 /
>> https://github.com/apache/spark/pull/22429
>> - https://issues.apache.org/jira/browse/SPARK-25429 /
>> https://github.com/apache/spark/pull/22420
>>
>>
>> 2017년 8월 3일 (목) 오전 9:06, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>
>>> I think it works for anyone who can leave a web link and comment.
>>>
>>>
>>> For "in progress" Resolution, it looks I am unable to manually set.
>>> Please let me know if anyone knows.
>>>
>>>
>>> For single JIRA, I manually modified the script to process single item
>>> before. I guess you know what the script does but want to rather test,
>>> right?
>>>
>>> This looks requiring Python JIRA client, which I installed by pip
>>> install jira==1.0.7 and I personally tested (and just double checked this)
>>> after commenting out JIRA related codes:
>>>
>>>
>>> diff --git a/dev/github_jira_sync.py b/dev/github_jira_sync.py
>>> index acc9aeabbb9..c8adf0c4ade 100755
>>> --- a/dev/github_jira_sync.py
>>> +++ b/dev/github_jira_sync.py
>>> @@ -101,8 +101,8 @@ def get_max_pr():
>>> return 0
>>>
>>>
>>> -jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>> - basic_auth=(JIRA_USERNAME,
>>> JIRA_PASSWORD))
>>> +# jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>> +# basic_auth=(JIRA_USERNAME,
>>> JIRA_PASSWORD))
>>>
>>> jira_prs = get_jira_prs()
>>>
>>> @@ -124,14 +124,15 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>> int(kv[1]['number'])):
>>> url = pr['html_url']
>>> title = "[Github] Pull Request #%s (%s)" % (pr['number'],
>>> pr['user']['login'])
>>> try:
>>> - existing_links = map(lambda l: l.raw['object']['url'],
>>> jira_client.remote_links(issue))
>>> + # existing_links = map(lambda l: l.raw['object']['url'],
>>> jira_client.remote_links(issue))
>>> + pass
>>> except:
>>> print("Failure reading JIRA %s (does it exist?)" % issue)
>>> print(sys.exc_info()[0])
>>> continue
>>>
>>> - if url in existing_links:
>>> - continue
>>> + # if url in existing_links:
>>> + # continue
>>>
>>> icon = {"title": "Pull request #%s" % pr['number'],
>>> "url16x16": "https://assets-cdn.github.com/favicon.ico"}
>>> @@ -139,12 +140,12 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>> int(kv[1]['number'])):
>>> # For all possible fields see:
>>> #
>>> https://developer.atlassian.com/display/JIRADEV/Fields+in+Remote+Issue+Links
>>> # application = {"name": "Github pull requests", "type":
>>> "org.apache.spark.jira.github"}
>>> - jira_client.add_remote_link(issue, destination)
>>> + # jira_client.add_remote_link(issue, destination)
>>>
>>> comment = "User '%s' has created a pull request for this issue:" %
>>> pr['user']['login']
>>> comment += "\n%s" % pr['html_url']
>>> - if pr_num >= MIN_COMMENT_PR:
>>> - jira_client.add_comment(issue, comment)
>>> + # if pr_num >= MIN_COMMENT_PR:
>>> + # jira_client.add_comment(issue, comment)
>>>
>>> print("Added link %s <-> PR #%s" % (issue, pr['number']))
>>> num_updates += 1
>>>
>>>
>>> and
>>>
>>>
>>> rm .github-jira-max
>>> ./dev/github_jira_sync.py
>>>
>>>
>>>
>>> 2017-08-03 5:20 GMT+09:00 Josh Rosen <jo...@databricks.com>:
>>>
>>>> Usually the backend of https://spark-prs.appspot.com does the linking
>>>> while processing PR update tasks. It appears that the site's connections to
>>>> JIRA have started failing:
>>>>
>>>> ConnectionError: ('Connection aborted.', HTTPException('Deadline
>>>> exceeded while waiting for HTTP response from URL:
>>>> https://issues.apache.org/jira/rest/api/2/serverInfo',))
>>>>
>>>> From Stackdriver's log-based metrics, I can spot that this problem
>>>> started around July 24th. We're already using a much-higher-than-default
>>>> URL fetch timeout, so it's possible that the problem is related to access
>>>> credentials, IP blocks, outdated client libraries, or something else.
>>>>
>>>> On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cu...@gmail.com> wrote:
>>>>
>>>>> Thanks Hyukjin! I didn't see your previous message.. It looks like
>>>>> your manual run worked pretty well for the JIRAs I'm following, the only
>>>>> thing is that it didn't mark them as "in progress", but that's not a big
>>>>> deal. Otherwise that helps until we can find out why it's not doing this
>>>>> automatically. I'm not familiar with that script, can anyone run it to
>>>>> apply to a single JIRA they are working on?
>>>>>
>>>>> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I was wondering about this too..
>>>>>>
>>>>>>
>>>>>> Yes, actually, I have been manually adding some links by resembling
>>>>>> the same steps in the script before.
>>>>>>
>>>>>> I was thinking it'd rather be nicer to run this manually once and
>>>>>> then I ran this against single JIRA
>>>>>>
>>>>>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show
>>>>>> how it looks like and check if there
>>>>>>
>>>>>> is any issue or objection just in case.
>>>>>>
>>>>>>
>>>>>> Will run this manually now once. I will revert all my action manually
>>>>>> if there is any issue by doing this.
>>>>>>
>>>>>>
>>>>>> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>>>>>>
>>>>>>> Hyukjin mentioned this here earlier today and had run it manually,
>>>>>>> but yeah I'm not sure where it normally runs or why it hasn't. Shane not
>>>>>>> sure if you're the person to ask?
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Devs,
>>>>>>>>
>>>>>>>> I've noticed a couple PRs recently have not been automatically
>>>>>>>> linked to the related JIRAs. This was one of mine (I linked it manually)
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen
>>>>>>>> it happen elsewhere. I think this is the script that does it, but it
>>>>>>>> hasn't been changed recently
>>>>>>>> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
>>>>>>>> Anyone else seen this or know what's going on?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bryan
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
Re: Some PRs not automatically linked to JIRAs
Posted by Hyukjin Kwon <gu...@gmail.com>.
Seems same thing is happening again.
For instance,
- https://issues.apache.org/jira/browse/SPARK-25440 /
https://github.com/apache/spark/pull/22429
- https://issues.apache.org/jira/browse/SPARK-25429 /
https://github.com/apache/spark/pull/22420
2017년 8월 3일 (목) 오전 9:06, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> I think it works for anyone who can leave a web link and comment.
>
>
> For "in progress" Resolution, it looks I am unable to manually set.
> Please let me know if anyone knows.
>
>
> For single JIRA, I manually modified the script to process single item
> before. I guess you know what the script does but want to rather test,
> right?
>
> This looks requiring Python JIRA client, which I installed by pip install
> jira==1.0.7 and I personally tested (and just double checked this) after
> commenting out JIRA related codes:
>
>
> diff --git a/dev/github_jira_sync.py b/dev/github_jira_sync.py
> index acc9aeabbb9..c8adf0c4ade 100755
> --- a/dev/github_jira_sync.py
> +++ b/dev/github_jira_sync.py
> @@ -101,8 +101,8 @@ def get_max_pr():
> return 0
>
>
> -jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
> - basic_auth=(JIRA_USERNAME, JIRA_PASSWORD))
> +# jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
> +# basic_auth=(JIRA_USERNAME,
> JIRA_PASSWORD))
>
> jira_prs = get_jira_prs()
>
> @@ -124,14 +124,15 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
> int(kv[1]['number'])):
> url = pr['html_url']
> title = "[Github] Pull Request #%s (%s)" % (pr['number'],
> pr['user']['login'])
> try:
> - existing_links = map(lambda l: l.raw['object']['url'],
> jira_client.remote_links(issue))
> + # existing_links = map(lambda l: l.raw['object']['url'],
> jira_client.remote_links(issue))
> + pass
> except:
> print("Failure reading JIRA %s (does it exist?)" % issue)
> print(sys.exc_info()[0])
> continue
>
> - if url in existing_links:
> - continue
> + # if url in existing_links:
> + # continue
>
> icon = {"title": "Pull request #%s" % pr['number'],
> "url16x16": "https://assets-cdn.github.com/favicon.ico"}
> @@ -139,12 +140,12 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
> int(kv[1]['number'])):
> # For all possible fields see:
> #
> https://developer.atlassian.com/display/JIRADEV/Fields+in+Remote+Issue+Links
> # application = {"name": "Github pull requests", "type":
> "org.apache.spark.jira.github"}
> - jira_client.add_remote_link(issue, destination)
> + # jira_client.add_remote_link(issue, destination)
>
> comment = "User '%s' has created a pull request for this issue:" %
> pr['user']['login']
> comment += "\n%s" % pr['html_url']
> - if pr_num >= MIN_COMMENT_PR:
> - jira_client.add_comment(issue, comment)
> + # if pr_num >= MIN_COMMENT_PR:
> + # jira_client.add_comment(issue, comment)
>
> print("Added link %s <-> PR #%s" % (issue, pr['number']))
> num_updates += 1
>
>
> and
>
>
> rm .github-jira-max
> ./dev/github_jira_sync.py
>
>
>
> 2017-08-03 5:20 GMT+09:00 Josh Rosen <jo...@databricks.com>:
>
>> Usually the backend of https://spark-prs.appspot.com does the linking
>> while processing PR update tasks. It appears that the site's connections to
>> JIRA have started failing:
>>
>> ConnectionError: ('Connection aborted.', HTTPException('Deadline exceeded
>> while waiting for HTTP response from URL:
>> https://issues.apache.org/jira/rest/api/2/serverInfo',))
>>
>> From Stackdriver's log-based metrics, I can spot that this problem
>> started around July 24th. We're already using a much-higher-than-default
>> URL fetch timeout, so it's possible that the problem is related to access
>> credentials, IP blocks, outdated client libraries, or something else.
>>
>> On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cu...@gmail.com> wrote:
>>
>>> Thanks Hyukjin! I didn't see your previous message.. It looks like
>>> your manual run worked pretty well for the JIRAs I'm following, the only
>>> thing is that it didn't mark them as "in progress", but that's not a big
>>> deal. Otherwise that helps until we can find out why it's not doing this
>>> automatically. I'm not familiar with that script, can anyone run it to
>>> apply to a single JIRA they are working on?
>>>
>>> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> I was wondering about this too..
>>>>
>>>>
>>>> Yes, actually, I have been manually adding some links by resembling the
>>>> same steps in the script before.
>>>>
>>>> I was thinking it'd rather be nicer to run this manually once and then
>>>> I ran this against single JIRA
>>>>
>>>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show how
>>>> it looks like and check if there
>>>>
>>>> is any issue or objection just in case.
>>>>
>>>>
>>>> Will run this manually now once. I will revert all my action manually
>>>> if there is any issue by doing this.
>>>>
>>>>
>>>> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>>>>
>>>>> Hyukjin mentioned this here earlier today and had run it manually, but
>>>>> yeah I'm not sure where it normally runs or why it hasn't. Shane not sure
>>>>> if you're the person to ask?
>>>>>
>>>>>
>>>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com> wrote:
>>>>>
>>>>>> Hi Devs,
>>>>>>
>>>>>> I've noticed a couple PRs recently have not been automatically linked
>>>>>> to the related JIRAs. This was one of mine (I linked it manually)
>>>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen it
>>>>>> happen elsewhere. I think this is the script that does it, but it hasn't
>>>>>> been changed recently
>>>>>> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
>>>>>> Anyone else seen this or know what's going on?
>>>>>>
>>>>>> Thanks,
>>>>>> Bryan
>>>>>>
>>>>>
>>>>
>>>
>
Re: Some PRs not automatically linked to JIRAs
Posted by Hyukjin Kwon <gu...@gmail.com>.
I think it works for anyone who can leave a web link and comment.
For "in progress" Resolution, it looks I am unable to manually set. Please
let me know if anyone knows.
For single JIRA, I manually modified the script to process single item
before. I guess you know what the script does but want to rather test,
right?
This looks requiring Python JIRA client, which I installed by pip install
jira==1.0.7 and I personally tested (and just double checked this) after
commenting out JIRA related codes:
diff --git a/dev/github_jira_sync.py b/dev/github_jira_sync.py
index acc9aeabbb9..c8adf0c4ade 100755
--- a/dev/github_jira_sync.py
+++ b/dev/github_jira_sync.py
@@ -101,8 +101,8 @@ def get_max_pr():
return 0
-jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
- basic_auth=(JIRA_USERNAME, JIRA_PASSWORD))
+# jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
+# basic_auth=(JIRA_USERNAME, JIRA_PASSWORD))
jira_prs = get_jira_prs()
@@ -124,14 +124,15 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
int(kv[1]['number'])):
url = pr['html_url']
title = "[Github] Pull Request #%s (%s)" % (pr['number'],
pr['user']['login'])
try:
- existing_links = map(lambda l: l.raw['object']['url'],
jira_client.remote_links(issue))
+ # existing_links = map(lambda l: l.raw['object']['url'],
jira_client.remote_links(issue))
+ pass
except:
print("Failure reading JIRA %s (does it exist?)" % issue)
print(sys.exc_info()[0])
continue
- if url in existing_links:
- continue
+ # if url in existing_links:
+ # continue
icon = {"title": "Pull request #%s" % pr['number'],
"url16x16": "https://assets-cdn.github.com/favicon.ico"}
@@ -139,12 +140,12 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
int(kv[1]['number'])):
# For all possible fields see:
#
https://developer.atlassian.com/display/JIRADEV/Fields+in+Remote+Issue+Links
# application = {"name": "Github pull requests", "type":
"org.apache.spark.jira.github"}
- jira_client.add_remote_link(issue, destination)
+ # jira_client.add_remote_link(issue, destination)
comment = "User '%s' has created a pull request for this issue:" %
pr['user']['login']
comment += "\n%s" % pr['html_url']
- if pr_num >= MIN_COMMENT_PR:
- jira_client.add_comment(issue, comment)
+ # if pr_num >= MIN_COMMENT_PR:
+ # jira_client.add_comment(issue, comment)
print("Added link %s <-> PR #%s" % (issue, pr['number']))
num_updates += 1
and
rm .github-jira-max
./dev/github_jira_sync.py
2017-08-03 5:20 GMT+09:00 Josh Rosen <jo...@databricks.com>:
> Usually the backend of https://spark-prs.appspot.com does the linking
> while processing PR update tasks. It appears that the site's connections to
> JIRA have started failing:
>
> ConnectionError: ('Connection aborted.', HTTPException('Deadline exceeded
> while waiting for HTTP response from URL: https://issues.apache.org/
> jira/rest/api/2/serverInfo',))
>
> From Stackdriver's log-based metrics, I can spot that this problem started
> around July 24th. We're already using a much-higher-than-default URL fetch
> timeout, so it's possible that the problem is related to access
> credentials, IP blocks, outdated client libraries, or something else.
>
> On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cu...@gmail.com> wrote:
>
>> Thanks Hyukjin! I didn't see your previous message.. It looks like your
>> manual run worked pretty well for the JIRAs I'm following, the only thing
>> is that it didn't mark them as "in progress", but that's not a big deal.
>> Otherwise that helps until we can find out why it's not doing this
>> automatically. I'm not familiar with that script, can anyone run it to
>> apply to a single JIRA they are working on?
>>
>> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com>
>> wrote:
>>
>>> I was wondering about this too..
>>>
>>>
>>> Yes, actually, I have been manually adding some links by resembling the
>>> same steps in the script before.
>>>
>>> I was thinking it'd rather be nicer to run this manually once and then I
>>> ran this against single JIRA
>>>
>>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show how
>>> it looks like and check if there
>>>
>>> is any issue or objection just in case.
>>>
>>>
>>> Will run this manually now once. I will revert all my action manually if
>>> there is any issue by doing this.
>>>
>>>
>>> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>>>
>>>> Hyukjin mentioned this here earlier today and had run it manually, but
>>>> yeah I'm not sure where it normally runs or why it hasn't. Shane not sure
>>>> if you're the person to ask?
>>>>
>>>>
>>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com> wrote:
>>>>
>>>>> Hi Devs,
>>>>>
>>>>> I've noticed a couple PRs recently have not been automatically linked
>>>>> to the related JIRAs. This was one of mine (I linked it manually)
>>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen it
>>>>> happen elsewhere. I think this is the script that does it, but it hasn't
>>>>> been changed recently https://github.com/apache/spark/blob/master/dev/
>>>>> github_jira_sync.py. Anyone else seen this or know what's going on?
>>>>>
>>>>> Thanks,
>>>>> Bryan
>>>>>
>>>>
>>>
>>
Re: Some PRs not automatically linked to JIRAs
Posted by Josh Rosen <jo...@databricks.com>.
Usually the backend of https://spark-prs.appspot.com does the linking while
processing PR update tasks. It appears that the site's connections to JIRA
have started failing:
ConnectionError: ('Connection aborted.', HTTPException('Deadline exceeded
while waiting for HTTP response from URL:
https://issues.apache.org/jira/rest/api/2/serverInfo',))
From Stackdriver's log-based metrics, I can spot that this problem started
around July 24th. We're already using a much-higher-than-default URL fetch
timeout, so it's possible that the problem is related to access
credentials, IP blocks, outdated client libraries, or something else.
On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cu...@gmail.com> wrote:
> Thanks Hyukjin! I didn't see your previous message.. It looks like your
> manual run worked pretty well for the JIRAs I'm following, the only thing
> is that it didn't mark them as "in progress", but that's not a big deal.
> Otherwise that helps until we can find out why it's not doing this
> automatically. I'm not familiar with that script, can anyone run it to
> apply to a single JIRA they are working on?
>
> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> I was wondering about this too..
>>
>>
>> Yes, actually, I have been manually adding some links by resembling the
>> same steps in the script before.
>>
>> I was thinking it'd rather be nicer to run this manually once and then I
>> ran this against single JIRA
>>
>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show how it
>> looks like and check if there
>>
>> is any issue or objection just in case.
>>
>>
>> Will run this manually now once. I will revert all my action manually if
>> there is any issue by doing this.
>>
>>
>> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>>
>>> Hyukjin mentioned this here earlier today and had run it manually, but
>>> yeah I'm not sure where it normally runs or why it hasn't. Shane not sure
>>> if you're the person to ask?
>>>
>>>
>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com> wrote:
>>>
>>>> Hi Devs,
>>>>
>>>> I've noticed a couple PRs recently have not been automatically linked
>>>> to the related JIRAs. This was one of mine (I linked it manually)
>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen it
>>>> happen elsewhere. I think this is the script that does it, but it hasn't
>>>> been changed recently
>>>> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
>>>> Anyone else seen this or know what's going on?
>>>>
>>>> Thanks,
>>>> Bryan
>>>>
>>>
>>
>
Re: Some PRs not automatically linked to JIRAs
Posted by Bryan Cutler <cu...@gmail.com>.
Thanks Hyukjin! I didn't see your previous message.. It looks like your
manual run worked pretty well for the JIRAs I'm following, the only thing
is that it didn't mark them as "in progress", but that's not a big deal.
Otherwise that helps until we can find out why it's not doing this
automatically. I'm not familiar with that script, can anyone run it to
apply to a single JIRA they are working on?
On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com> wrote:
> I was wondering about this too..
>
>
> Yes, actually, I have been manually adding some links by resembling the
> same steps in the script before.
>
> I was thinking it'd rather be nicer to run this manually once and then I
> ran this against single JIRA
>
> first - https://issues.apache.org/jira/browse/SPARK-21526 to show how it
> looks like and check if there
>
> is any issue or objection just in case.
>
>
> Will run this manually now once. I will revert all my action manually if
> there is any issue by doing this.
>
>
> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>
>> Hyukjin mentioned this here earlier today and had run it manually, but
>> yeah I'm not sure where it normally runs or why it hasn't. Shane not sure
>> if you're the person to ask?
>>
>>
>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com> wrote:
>>
>>> Hi Devs,
>>>
>>> I've noticed a couple PRs recently have not been automatically linked to
>>> the related JIRAs. This was one of mine (I linked it manually)
>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen it
>>> happen elsewhere. I think this is the script that does it, but it hasn't
>>> been changed recently https://github.com/ap
>>> ache/spark/blob/master/dev/github_jira_sync.py. Anyone else seen this
>>> or know what's going on?
>>>
>>> Thanks,
>>> Bryan
>>>
>>
>
Re: Some PRs not automatically linked to JIRAs
Posted by Hyukjin Kwon <gu...@gmail.com>.
I was wondering about this too..
Yes, actually, I have been manually adding some links by resembling the
same steps in the script before.
I was thinking it'd rather be nicer to run this manually once and then I
ran this against single JIRA
first - https://issues.apache.org/jira/browse/SPARK-21526 to show how it
looks like and check if there
is any issue or objection just in case.
Will run this manually now once. I will revert all my action manually if
there is any issue by doing this.
2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
> Hyukjin mentioned this here earlier today and had run it manually, but
> yeah I'm not sure where it normally runs or why it hasn't. Shane not sure
> if you're the person to ask?
>
>
> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com> wrote:
>
>> Hi Devs,
>>
>> I've noticed a couple PRs recently have not been automatically linked to
>> the related JIRAs. This was one of mine (I linked it manually)
>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen it
>> happen elsewhere. I think this is the script that does it, but it hasn't
>> been changed recently https://github.com/apache/spark/blob/master/dev/git
>> hub_jira_sync.py. Anyone else seen this or know what's going on?
>>
>> Thanks,
>> Bryan
>>
>
Re: Some PRs not automatically linked to JIRAs
Posted by Sean Owen <so...@cloudera.com>.
Hyukjin mentioned this here earlier today and had run it manually, but yeah
I'm not sure where it normally runs or why it hasn't. Shane not sure if
you're the person to ask?
On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com> wrote:
> Hi Devs,
>
> I've noticed a couple PRs recently have not been automatically linked to
> the related JIRAs. This was one of mine (I linked it manually)
> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen it
> happen elsewhere. I think this is the script that does it, but it hasn't
> been changed recently
> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
> Anyone else seen this or know what's going on?
>
> Thanks,
> Bryan
>