You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@gmail.com> on 2018/11/21 08:49:02 UTC
Re: Some PRs not automatically linked to JIRAs
This issue is still persistent.
https://issues.apache.org/jira/browse/SPARK-26132
https://issues.apache.org/jira/browse/SPARK-26129
https://issues.apache.org/jira/browse/SPARK-26127
https://issues.apache.org/jira/browse/SPARK-26109
https://issues.apache.org/jira/browse/SPARK-26106
https://issues.apache.org/jira/browse/SPARK-26104
the links are still duplicated. Looks the scripts are being ran at multiple
places.
2018년 10월 30일 (화) 오후 4:58, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> Duplicated link problem looks still persistent:
>
> https://issues.apache.org/jira/browse/SPARK-25881
> https://issues.apache.org/jira/browse/SPARK-25880
>
> I suspect if there are two places that runs this script. Not a big deal
> but people that can fix this are specific.
> I am leaving another reminder here.
>
> Thanks.
>
>
> 2018년 10월 1일 (월) 오후 7:15, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>
>> Seems fixed but looks it starts to leave duplicated PR links for some
>> recent JIRAs. Not a big deal but are they being ran in multiple places
>> maybe?
>>
>> For instance,
>>
>> https://issues.apache.org/jira/browse/SPARK-25579
>> https://issues.apache.org/jira/browse/SPARK-25574
>> https://issues.apache.org/jira/browse/SPARK-25564
>>
>>
>> 2018년 9월 17일 (월) 오후 10:09, Ilan Filonenko <il...@gmail.com>님이 작성:
>>
>>> Same over here:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-25291 /
>>> https://github.com/apache/spark/pull/22415
>>>
>>> On Sun, Sep 16, 2018 at 10:09 PM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> Seems same thing is happening again.
>>>>
>>>> For instance,
>>>> - https://issues.apache.org/jira/browse/SPARK-25440 /
>>>> https://github.com/apache/spark/pull/22429
>>>> - https://issues.apache.org/jira/browse/SPARK-25429 /
>>>> https://github.com/apache/spark/pull/22420
>>>>
>>>>
>>>> 2017년 8월 3일 (목) 오전 9:06, Hyukjin Kwon <gu...@gmail.com>님이 작성:
>>>>
>>>>> I think it works for anyone who can leave a web link and comment.
>>>>>
>>>>>
>>>>> For "in progress" Resolution, it looks I am unable to manually set.
>>>>> Please let me know if anyone knows.
>>>>>
>>>>>
>>>>> For single JIRA, I manually modified the script to process single item
>>>>> before. I guess you know what the script does but want to rather test,
>>>>> right?
>>>>>
>>>>> This looks requiring Python JIRA client, which I installed by pip
>>>>> install jira==1.0.7 and I personally tested (and just double checked this)
>>>>> after commenting out JIRA related codes:
>>>>>
>>>>>
>>>>> diff --git a/dev/github_jira_sync.py b/dev/github_jira_sync.py
>>>>> index acc9aeabbb9..c8adf0c4ade 100755
>>>>> --- a/dev/github_jira_sync.py
>>>>> +++ b/dev/github_jira_sync.py
>>>>> @@ -101,8 +101,8 @@ def get_max_pr():
>>>>> return 0
>>>>>
>>>>>
>>>>> -jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>>>> - basic_auth=(JIRA_USERNAME,
>>>>> JIRA_PASSWORD))
>>>>> +# jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>>>> +# basic_auth=(JIRA_USERNAME,
>>>>> JIRA_PASSWORD))
>>>>>
>>>>> jira_prs = get_jira_prs()
>>>>>
>>>>> @@ -124,14 +124,15 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>>>> int(kv[1]['number'])):
>>>>> url = pr['html_url']
>>>>> title = "[Github] Pull Request #%s (%s)" % (pr['number'],
>>>>> pr['user']['login'])
>>>>> try:
>>>>> - existing_links = map(lambda l: l.raw['object']['url'],
>>>>> jira_client.remote_links(issue))
>>>>> + # existing_links = map(lambda l: l.raw['object']['url'],
>>>>> jira_client.remote_links(issue))
>>>>> + pass
>>>>> except:
>>>>> print("Failure reading JIRA %s (does it exist?)" % issue)
>>>>> print(sys.exc_info()[0])
>>>>> continue
>>>>>
>>>>> - if url in existing_links:
>>>>> - continue
>>>>> + # if url in existing_links:
>>>>> + # continue
>>>>>
>>>>> icon = {"title": "Pull request #%s" % pr['number'],
>>>>> "url16x16": "https://assets-cdn.github.com/favicon.ico"}
>>>>> @@ -139,12 +140,12 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>>>> int(kv[1]['number'])):
>>>>> # For all possible fields see:
>>>>> #
>>>>> https://developer.atlassian.com/display/JIRADEV/Fields+in+Remote+Issue+Links
>>>>> # application = {"name": "Github pull requests", "type":
>>>>> "org.apache.spark.jira.github"}
>>>>> - jira_client.add_remote_link(issue, destination)
>>>>> + # jira_client.add_remote_link(issue, destination)
>>>>>
>>>>> comment = "User '%s' has created a pull request for this issue:"
>>>>> % pr['user']['login']
>>>>> comment += "\n%s" % pr['html_url']
>>>>> - if pr_num >= MIN_COMMENT_PR:
>>>>> - jira_client.add_comment(issue, comment)
>>>>> + # if pr_num >= MIN_COMMENT_PR:
>>>>> + # jira_client.add_comment(issue, comment)
>>>>>
>>>>> print("Added link %s <-> PR #%s" % (issue, pr['number']))
>>>>> num_updates += 1
>>>>>
>>>>>
>>>>> and
>>>>>
>>>>>
>>>>> rm .github-jira-max
>>>>> ./dev/github_jira_sync.py
>>>>>
>>>>>
>>>>>
>>>>> 2017-08-03 5:20 GMT+09:00 Josh Rosen <jo...@databricks.com>:
>>>>>
>>>>>> Usually the backend of https://spark-prs.appspot.com does the
>>>>>> linking while processing PR update tasks. It appears that the site's
>>>>>> connections to JIRA have started failing:
>>>>>>
>>>>>> ConnectionError: ('Connection aborted.', HTTPException('Deadline
>>>>>> exceeded while waiting for HTTP response from URL:
>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo',))
>>>>>>
>>>>>> From Stackdriver's log-based metrics, I can spot that this problem
>>>>>> started around July 24th. We're already using a much-higher-than-default
>>>>>> URL fetch timeout, so it's possible that the problem is related to access
>>>>>> credentials, IP blocks, outdated client libraries, or something else.
>>>>>>
>>>>>> On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Hyukjin! I didn't see your previous message.. It looks like
>>>>>>> your manual run worked pretty well for the JIRAs I'm following, the only
>>>>>>> thing is that it didn't mark them as "in progress", but that's not a big
>>>>>>> deal. Otherwise that helps until we can find out why it's not doing this
>>>>>>> automatically. I'm not familiar with that script, can anyone run it to
>>>>>>> apply to a single JIRA they are working on?
>>>>>>>
>>>>>>> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I was wondering about this too..
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, actually, I have been manually adding some links by resembling
>>>>>>>> the same steps in the script before.
>>>>>>>>
>>>>>>>> I was thinking it'd rather be nicer to run this manually once and
>>>>>>>> then I ran this against single JIRA
>>>>>>>>
>>>>>>>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show
>>>>>>>> how it looks like and check if there
>>>>>>>>
>>>>>>>> is any issue or objection just in case.
>>>>>>>>
>>>>>>>>
>>>>>>>> Will run this manually now once. I will revert all my action
>>>>>>>> manually if there is any issue by doing this.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2017-08-03 3:50 GMT+09:00 Sean Owen <so...@cloudera.com>:
>>>>>>>>
>>>>>>>>> Hyukjin mentioned this here earlier today and had run it manually,
>>>>>>>>> but yeah I'm not sure where it normally runs or why it hasn't. Shane not
>>>>>>>>> sure if you're the person to ask?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Devs,
>>>>>>>>>>
>>>>>>>>>> I've noticed a couple PRs recently have not been automatically
>>>>>>>>>> linked to the related JIRAs. This was one of mine (I linked it manually)
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've seen
>>>>>>>>>> it happen elsewhere. I think this is the script that does it, but it
>>>>>>>>>> hasn't been changed recently
>>>>>>>>>> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
>>>>>>>>>> Anyone else seen this or know what's going on?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bryan
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>