You are viewing a plain text version of this content. The canonical link for it is here.

Posted to builds@apache.org by Ismael Juma <is...@juma.me.uk> on 2017/04/27 10:16:39 UTC

Issues with Kafka PR builds since Jenkins emergency restart

Hi,

As explained in INFRA-14011[1], we're having issues with Kafka PR builds
since the recent Jenkins restart.

Builds hang for a long time with the message:

"GitHub pull request #2301 of commit
0cc900288e61b05b4095b63d28969e16167e5d88, no merge conflicts."[2]

This is problematic for Kafka (we generally only merge PRs when builds are
green), but also for other projects because these builds are taking a lot
of executor slots for long periods of time and not doing anything.

In one of the builds that I interrupted, there were some errors related to
API rate limits:

Unable to query GitHub for status of
PullRequestjava.io.InterruptedIOException
at org.kohsuke.github.RateLimitHandler$1.onError(RateLimitHandler.java:41)
at org.kohsuke.github.Requester.handleApiError(Requester.java:666)
at org.kohsuke.github.Requester._to(Requester.java:283)
at org.kohsuke.github.Requester.to(Requester.java:231)
at org.kohsuke.github.GHPullRequest.populate(GHPullRequest.java:207)
at org.kohsuke.github.GHPullRequest.getMergeable(GHPullRequest.java:170)
at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(GhprbBuilds.java:101)
at
org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(GhprbBuildListener.java:18)
at hudson.model.listeners.RunListener.fireStarted(RunListener.java:240)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
exceeded for asfbot.","documentation_url":"
https://developer.github.com/v3/#rate-limiting"}
at org.kohsuke.github.Requester.handleApiError(Requester.java:654)
... 11 more
Caused by: org.kohsuke.github.HttpException: Server returned HTTP response
code: 403, message: 'Forbidden' for URL:
https://api.github.com/repos/apache/kafka/pulls/2910
at org.kohsuke.github.Requester.parse(Requester.java:612)
at org.kohsuke.github.Requester._to(Requester.java:262)
... 10 more
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL: https://api.github.com/repos/apache/kafka/pulls/2910
at sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.kohsuke.github.Requester.parse(Requester.java:596)
... 11 more
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL: https://api.github.com/repos/apache/kafka/pulls/2910
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1876)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.kohsuke.github.Requester.parse(Requester.java:586)
... 11 more
Setting status of b5df4d319bc65e847a8c18ad20660f3e506e9a16 to PENDING with
url https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/ and
message: 'Build started sha1 is original commit.'[3]

Could this be a hint of the underlying issue?

These problems are affecting the following jobs:

https://builds.apache.org/job/kafka-pr-jdk7-scala2.10
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12

Thanks for your help.

Ismael

[1] https://issues.apache.org/jira/browse/INFRA-14011
[2] https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3201/console
[3] https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/console

Re: Issues with Kafka PR builds since Jenkins emergency restart

Posted by Ismael Juma <is...@juma.me.uk>.

Everything is back to normal.

Thanks,
Ismael

On Thu, Apr 27, 2017 at 1:37 PM, Ismael Juma <is...@juma.me.uk> wrote:

> Thanks!
>
> Ismael
>
> On Thu, Apr 27, 2017 at 11:53 AM, Gavin McDonald <ga...@16degrees.com.au>
> wrote:
>
>> Hi,
>>
>> Yes, we have contacted Github Support about the rate limit, and are yet
>> to hear back.
>>
>> Will let you know when we hear something.
>>
>> Gav… (ASF Infra)
>>
>> > On 27 Apr 2017, at 8:16 pm, Ismael Juma <is...@juma.me.uk> wrote:
>> >
>> > Hi,
>> >
>> > As explained in INFRA-14011[1], we're having issues with Kafka PR builds
>> > since the recent Jenkins restart.
>> >
>> > Builds hang for a long time with the message:
>> >
>> > "GitHub pull request #2301 of commit
>> > 0cc900288e61b05b4095b63d28969e16167e5d88, no merge conflicts."[2]
>> >
>> > This is problematic for Kafka (we generally only merge PRs when builds
>> are
>> > green), but also for other projects because these builds are taking a
>> lot
>> > of executor slots for long periods of time and not doing anything.
>> >
>> > In one of the builds that I interrupted, there were some errors related
>> to
>> > API rate limits:
>> >
>> > Unable to query GitHub for status of
>> > PullRequestjava.io.InterruptedIOException
>> > at org.kohsuke.github.RateLimitHandler$1.onError(RateLimitHandl
>> er.java:41)
>> > at org.kohsuke.github.Requester.handleApiError(Requester.java:666)
>> > at org.kohsuke.github.Requester._to(Requester.java:283)
>> > at org.kohsuke.github.Requester.to(Requester.java:231)
>> > at org.kohsuke.github.GHPullRequest.populate(GHPullRequest.java:207)
>> > at org.kohsuke.github.GHPullRequest.getMergeable(GHPullRequest.
>> java:170)
>> > at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(GhprbBuild
>> s.java:101)
>> > at
>> > org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(Ghp
>> rbBuildListener.java:18)
>> > at hudson.model.listeners.RunListener.fireStarted(RunListener.java:240)
>> > at hudson.model.Run.execute(Run.java:1724)
>> > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>> > at hudson.model.ResourceController.execute(ResourceController.java:98)
>> > at hudson.model.Executor.run(Executor.java:404)
>> > Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
>> > exceeded for asfbot.","documentation_url":"
>> > https://developer.github.com/v3/#rate-limiting"}
>> > at org.kohsuke.github.Requester.handleApiError(Requester.java:654)
>> > ... 11 more
>> > Caused by: org.kohsuke.github.HttpException: Server returned HTTP
>> response
>> > code: 403, message: 'Forbidden' for URL:
>> > https://api.github.com/repos/apache/kafka/pulls/2910
>> > at org.kohsuke.github.Requester.parse(Requester.java:612)
>> > at org.kohsuke.github.Requester._to(Requester.java:262)
>> > ... 10 more
>> > Caused by: java.io.IOException: Server returned HTTP response code: 403
>> for
>> > URL: https://api.github.com/repos/apache/kafka/pulls/2910
>> > at sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown
>> Source)
>> > at
>> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>> legatingConstructorAccessorImpl.java:45)
>> > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> > at
>> > sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLCo
>> nnection.java:1926)
>> > at
>> > sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLCo
>> nnection.java:1921)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at
>> > sun.net.www.protocol.http.HttpURLConnection.getChainedExcept
>> ion(HttpURLConnection.java:1920)
>> > at
>> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
>> HttpURLConnection.java:1490)
>> > at
>> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(H
>> ttpURLConnection.java:1474)
>> > at
>> > sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputSt
>> ream(HttpsURLConnectionImpl.java:254)
>> > at org.kohsuke.github.Requester.parse(Requester.java:596)
>> > ... 11 more
>> > Caused by: java.io.IOException: Server returned HTTP response code: 403
>> for
>> > URL: https://api.github.com/repos/apache/kafka/pulls/2910
>> > at
>> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
>> HttpURLConnection.java:1876)
>> > at
>> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(H
>> ttpURLConnection.java:1474)
>> > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection
>> .java:480)
>> > at
>> > sun.net.www.protocol.https.HttpsURLConnectionImpl.getRespons
>> eCode(HttpsURLConnectionImpl.java:338)
>> > at org.kohsuke.github.Requester.parse(Requester.java:586)
>> > ... 11 more
>> > Setting status of b5df4d319bc65e847a8c18ad20660f3e506e9a16 to PENDING
>> with
>> > url https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/ and
>> > message: 'Build started sha1 is original commit.'[3]
>> >
>> > Could this be a hint of the underlying issue?
>> >
>> > These problems are affecting the following jobs:
>> >
>> > https://builds.apache.org/job/kafka-pr-jdk7-scala2.10
>> > https://builds.apache.org/job/kafka-pr-jdk8-scala2.11
>> > https://builds.apache.org/job/kafka-pr-jdk8-scala2.12
>> >
>> > Thanks for your help.
>> >
>> > Ismael
>> >
>> > [1] https://issues.apache.org/jira/browse/INFRA-14011
>> > [2] https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3201/console
>> > [3] https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/console
>>
>>
>

Re: Issues with Kafka PR builds since Jenkins emergency restart

Posted by Ismael Juma <is...@juma.me.uk>.

Thanks!

Ismael

On Thu, Apr 27, 2017 at 11:53 AM, Gavin McDonald <ga...@16degrees.com.au>
wrote:

> Hi,
>
> Yes, we have contacted Github Support about the rate limit, and are yet to
> hear back.
>
> Will let you know when we hear something.
>
> Gav… (ASF Infra)
>
> > On 27 Apr 2017, at 8:16 pm, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > Hi,
> >
> > As explained in INFRA-14011[1], we're having issues with Kafka PR builds
> > since the recent Jenkins restart.
> >
> > Builds hang for a long time with the message:
> >
> > "GitHub pull request #2301 of commit
> > 0cc900288e61b05b4095b63d28969e16167e5d88, no merge conflicts."[2]
> >
> > This is problematic for Kafka (we generally only merge PRs when builds
> are
> > green), but also for other projects because these builds are taking a lot
> > of executor slots for long periods of time and not doing anything.
> >
> > In one of the builds that I interrupted, there were some errors related
> to
> > API rate limits:
> >
> > Unable to query GitHub for status of
> > PullRequestjava.io.InterruptedIOException
> > at org.kohsuke.github.RateLimitHandler$1.onError(
> RateLimitHandler.java:41)
> > at org.kohsuke.github.Requester.handleApiError(Requester.java:666)
> > at org.kohsuke.github.Requester._to(Requester.java:283)
> > at org.kohsuke.github.Requester.to(Requester.java:231)
> > at org.kohsuke.github.GHPullRequest.populate(GHPullRequest.java:207)
> > at org.kohsuke.github.GHPullRequest.getMergeable(GHPullRequest.java:170)
> > at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(
> GhprbBuilds.java:101)
> > at
> > org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(
> GhprbBuildListener.java:18)
> > at hudson.model.listeners.RunListener.fireStarted(RunListener.java:240)
> > at hudson.model.Run.execute(Run.java:1724)
> > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> > at hudson.model.ResourceController.execute(ResourceController.java:98)
> > at hudson.model.Executor.run(Executor.java:404)
> > Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
> > exceeded for asfbot.","documentation_url":"
> > https://developer.github.com/v3/#rate-limiting"}
> > at org.kohsuke.github.Requester.handleApiError(Requester.java:654)
> > ... 11 more
> > Caused by: org.kohsuke.github.HttpException: Server returned HTTP
> response
> > code: 403, message: 'Forbidden' for URL:
> > https://api.github.com/repos/apache/kafka/pulls/2910
> > at org.kohsuke.github.Requester.parse(Requester.java:612)
> > at org.kohsuke.github.Requester._to(Requester.java:262)
> > ... 10 more
> > Caused by: java.io.IOException: Server returned HTTP response code: 403
> for
> > URL: https://api.github.com/repos/apache/kafka/pulls/2910
> > at sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown
> Source)
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > at
> > sun.net.www.protocol.http.HttpURLConnection$10.run(
> HttpURLConnection.java:1926)
> > at
> > sun.net.www.protocol.http.HttpURLConnection$10.run(
> HttpURLConnection.java:1921)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getChainedException(
> HttpURLConnection.java:1920)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
> HttpURLConnection.java:1490)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> HttpURLConnection.java:1474)
> > at
> > sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(
> HttpsURLConnectionImpl.java:254)
> > at org.kohsuke.github.Requester.parse(Requester.java:596)
> > ... 11 more
> > Caused by: java.io.IOException: Server returned HTTP response code: 403
> for
> > URL: https://api.github.com/repos/apache/kafka/pulls/2910
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
> HttpURLConnection.java:1876)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> HttpURLConnection.java:1474)
> > at java.net.HttpURLConnection.getResponseCode(
> HttpURLConnection.java:480)
> > at
> > sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(
> HttpsURLConnectionImpl.java:338)
> > at org.kohsuke.github.Requester.parse(Requester.java:586)
> > ... 11 more
> > Setting status of b5df4d319bc65e847a8c18ad20660f3e506e9a16 to PENDING
> with
> > url https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/ and
> > message: 'Build started sha1 is original commit.'[3]
> >
> > Could this be a hint of the underlying issue?
> >
> > These problems are affecting the following jobs:
> >
> > https://builds.apache.org/job/kafka-pr-jdk7-scala2.10
> > https://builds.apache.org/job/kafka-pr-jdk8-scala2.11
> > https://builds.apache.org/job/kafka-pr-jdk8-scala2.12
> >
> > Thanks for your help.
> >
> > Ismael
> >
> > [1] https://issues.apache.org/jira/browse/INFRA-14011
> > [2] https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3201/console
> > [3] https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/console
>
>

Re: Issues with Kafka PR builds since Jenkins emergency restart

Posted by Gavin McDonald <ga...@16degrees.com.au>.

Hi,

Yes, we have contacted Github Support about the rate limit, and are yet to hear back.

Will let you know when we hear something.

Gav… (ASF Infra)

> On 27 Apr 2017, at 8:16 pm, Ismael Juma <is...@juma.me.uk> wrote:
> 
> Hi,
> 
> As explained in INFRA-14011[1], we're having issues with Kafka PR builds
> since the recent Jenkins restart.
> 
> Builds hang for a long time with the message:
> 
> "GitHub pull request #2301 of commit
> 0cc900288e61b05b4095b63d28969e16167e5d88, no merge conflicts."[2]
> 
> This is problematic for Kafka (we generally only merge PRs when builds are
> green), but also for other projects because these builds are taking a lot
> of executor slots for long periods of time and not doing anything.
> 
> In one of the builds that I interrupted, there were some errors related to
> API rate limits:
> 
> Unable to query GitHub for status of
> PullRequestjava.io.InterruptedIOException
> at org.kohsuke.github.RateLimitHandler$1.onError(RateLimitHandler.java:41)
> at org.kohsuke.github.Requester.handleApiError(Requester.java:666)
> at org.kohsuke.github.Requester._to(Requester.java:283)
> at org.kohsuke.github.Requester.to(Requester.java:231)
> at org.kohsuke.github.GHPullRequest.populate(GHPullRequest.java:207)
> at org.kohsuke.github.GHPullRequest.getMergeable(GHPullRequest.java:170)
> at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(GhprbBuilds.java:101)
> at
> org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(GhprbBuildListener.java:18)
> at hudson.model.listeners.RunListener.fireStarted(RunListener.java:240)
> at hudson.model.Run.execute(Run.java:1724)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(ResourceController.java:98)
> at hudson.model.Executor.run(Executor.java:404)
> Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
> exceeded for asfbot.","documentation_url":"
> https://developer.github.com/v3/#rate-limiting"}
> at org.kohsuke.github.Requester.handleApiError(Requester.java:654)
> ... 11 more
> Caused by: org.kohsuke.github.HttpException: Server returned HTTP response
> code: 403, message: 'Forbidden' for URL:
> https://api.github.com/repos/apache/kafka/pulls/2910
> at org.kohsuke.github.Requester.parse(Requester.java:612)
> at org.kohsuke.github.Requester._to(Requester.java:262)
> ... 10 more
> Caused by: java.io.IOException: Server returned HTTP response code: 403 for
> URL: https://api.github.com/repos/apache/kafka/pulls/2910
> at sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
> at
> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at org.kohsuke.github.Requester.parse(Requester.java:596)
> ... 11 more
> Caused by: java.io.IOException: Server returned HTTP response code: 403 for
> URL: https://api.github.com/repos/apache/kafka/pulls/2910
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1876)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
> at org.kohsuke.github.Requester.parse(Requester.java:586)
> ... 11 more
> Setting status of b5df4d319bc65e847a8c18ad20660f3e506e9a16 to PENDING with
> url https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/ and
> message: 'Build started sha1 is original commit.'[3]
> 
> Could this be a hint of the underlying issue?
> 
> These problems are affecting the following jobs:
> 
> https://builds.apache.org/job/kafka-pr-jdk7-scala2.10
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.11
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.12
> 
> Thanks for your help.
> 
> Ismael
> 
> [1] https://issues.apache.org/jira/browse/INFRA-14011
> [2] https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3201/console
> [3] https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/console