You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Zoran Regvart <zo...@regvart.com> on 2021/02/17 09:27:25 UTC
Hitting GitHub API limits
Hi builders,
for the Apache Camel website build[1] we on occasion hit GitHub API
limits. If I'm not mistaken, the limit for unauthenticated API access
is accrued based on the source IP address, which I'm guessing is
shared across all projects accessing GitHub API from the same set of
Jenkins nodes.
We use the GitHub API to fetch user information (full name, avatar),
issues per milestone, and similar. The requests originate from our
static site generator - Hugo, and are cached for 12h[2].
Unfortunately, Hugo doesn't support setting any HTTP request headers,
and query based authentication will be discontinued[3]. So I don't see
a way of authenticating against the API in our case.
I'm considering our options, and I'd be happy to hear other
suggestions, and it seems that one option would be to run the build on
GitHub Actions. As far as I understand, there is a requirement to
build the websites on the ASF CI infrastructure (Jenkins, ...),
partially based on the service account setup that allows pushing to
the `asf-site` branch from there. We could replicate that setup on
GitHub Actions by having INFRA add a $GITHUB_TOKEN secret with push
permissions to the website git repository.
What in the way of precedent or applicable policy is there for this?
zoran
[1] https://ci-builds.apache.org/job/Camel/job/Camel.website/
[2] https://github.com/apache/camel-website/blob/master/config.toml#L24-L26
[3] https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/
--
Zoran Regvart
Re: Hitting GitHub API limits
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just one comment - you might want to take a look at
https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status.
documents which describes some of the open issues we have with GitHub
Actions - they relate to scalability/performance and security.
Also - maybe the IP Addresses of ASF infrastructure could have higher
limits (or even no limits) - maybe somehow infra could reach out to GitHub
and check? This seems like a super-simple solution if they could help us
with it.
I know DockerHub for example exempts apache images from pull rate limits
thanks to agreement with INFRA.
J.
On Wed, Feb 17, 2021 at 10:27 AM Zoran Regvart <zo...@regvart.com> wrote:
> Hi builders,
> for the Apache Camel website build[1] we on occasion hit GitHub API
> limits. If I'm not mistaken, the limit for unauthenticated API access
> is accrued based on the source IP address, which I'm guessing is
> shared across all projects accessing GitHub API from the same set of
> Jenkins nodes.
>
> We use the GitHub API to fetch user information (full name, avatar),
> issues per milestone, and similar. The requests originate from our
> static site generator - Hugo, and are cached for 12h[2].
>
> Unfortunately, Hugo doesn't support setting any HTTP request headers,
> and query based authentication will be discontinued[3]. So I don't see
> a way of authenticating against the API in our case.
>
> I'm considering our options, and I'd be happy to hear other
> suggestions, and it seems that one option would be to run the build on
> GitHub Actions. As far as I understand, there is a requirement to
> build the websites on the ASF CI infrastructure (Jenkins, ...),
> partially based on the service account setup that allows pushing to
> the `asf-site` branch from there. We could replicate that setup on
> GitHub Actions by having INFRA add a $GITHUB_TOKEN secret with push
> permissions to the website git repository.
>
> What in the way of precedent or applicable policy is there for this?
>
> zoran
>
> [1] https://ci-builds.apache.org/job/Camel/job/Camel.website/
> [2]
> https://github.com/apache/camel-website/blob/master/config.toml#L24-L26
> [3]
> https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/
> --
> Zoran Regvart
>
--
+48 660 796 129
Re: Hitting GitHub API limits
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just one comment - you might want to take a look at
https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status.
documents which describes some of the open issues we have with GitHub
Actions - they relate to scalability/performance and security.
Also - maybe the IP Addresses of ASF infrastructure could have higher
limits (or even no limits) - maybe somehow infra could reach out to GitHub
and check? This seems like a super-simple solution if they could help us
with it.
I know DockerHub for example exempts apache images from pull rate limits
thanks to agreement with INFRA.
J.
On Wed, Feb 17, 2021 at 10:27 AM Zoran Regvart <zo...@regvart.com> wrote:
> Hi builders,
> for the Apache Camel website build[1] we on occasion hit GitHub API
> limits. If I'm not mistaken, the limit for unauthenticated API access
> is accrued based on the source IP address, which I'm guessing is
> shared across all projects accessing GitHub API from the same set of
> Jenkins nodes.
>
> We use the GitHub API to fetch user information (full name, avatar),
> issues per milestone, and similar. The requests originate from our
> static site generator - Hugo, and are cached for 12h[2].
>
> Unfortunately, Hugo doesn't support setting any HTTP request headers,
> and query based authentication will be discontinued[3]. So I don't see
> a way of authenticating against the API in our case.
>
> I'm considering our options, and I'd be happy to hear other
> suggestions, and it seems that one option would be to run the build on
> GitHub Actions. As far as I understand, there is a requirement to
> build the websites on the ASF CI infrastructure (Jenkins, ...),
> partially based on the service account setup that allows pushing to
> the `asf-site` branch from there. We could replicate that setup on
> GitHub Actions by having INFRA add a $GITHUB_TOKEN secret with push
> permissions to the website git repository.
>
> What in the way of precedent or applicable policy is there for this?
>
> zoran
>
> [1] https://ci-builds.apache.org/job/Camel/job/Camel.website/
> [2]
> https://github.com/apache/camel-website/blob/master/config.toml#L24-L26
> [3]
> https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/
> --
> Zoran Regvart
>
--
+48 660 796 129
Re: Hitting GitHub API limits
Posted by David Jencks <da...@gmail.com>.
I’m 99% sure that infra is fine with building a site using GH actions as long as the authentication that pushes to the published GitHub repo is completely appropriate. IIUC Camel uses the website playbook project to publish to, so that part is easy, the auth token is the one for the current repo.
David Jencks
> On Feb 17, 2021, at 1:27 AM, Zoran Regvart <zo...@regvart.com> wrote:
>
> Hi builders,
> for the Apache Camel website build[1] we on occasion hit GitHub API
> limits. If I'm not mistaken, the limit for unauthenticated API access
> is accrued based on the source IP address, which I'm guessing is
> shared across all projects accessing GitHub API from the same set of
> Jenkins nodes.
>
> We use the GitHub API to fetch user information (full name, avatar),
> issues per milestone, and similar. The requests originate from our
> static site generator - Hugo, and are cached for 12h[2].
>
> Unfortunately, Hugo doesn't support setting any HTTP request headers,
> and query based authentication will be discontinued[3]. So I don't see
> a way of authenticating against the API in our case.
>
> I'm considering our options, and I'd be happy to hear other
> suggestions, and it seems that one option would be to run the build on
> GitHub Actions. As far as I understand, there is a requirement to
> build the websites on the ASF CI infrastructure (Jenkins, ...),
> partially based on the service account setup that allows pushing to
> the `asf-site` branch from there. We could replicate that setup on
> GitHub Actions by having INFRA add a $GITHUB_TOKEN secret with push
> permissions to the website git repository.
>
> What in the way of precedent or applicable policy is there for this?
>
> zoran
>
> [1] https://ci-builds.apache.org/job/Camel/job/Camel.website/
> [2] https://github.com/apache/camel-website/blob/master/config.toml#L24-L26
> [3] https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/
> --
> Zoran Regvart