You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by "Eric Friedrich (efriedri)" <ef...@cisco.com.INVALID> on 2018/06/15 17:26:39 UTC
PR to improve build process
I’ve opened a PR to produce Trafficserver and astats RPMs in the Traffic Control build process. Previously, new users would need to create their own build environments or ask for RPMS on Slack.
This PR will lower the barrier to entry for new users. For more involved users, theres now a set of scripts that will build inside a Dockerized Jenkins environment.
The changes rely on having all TS changes in a single git repository to form the RPM source tarball. I understand there are some users which have private tooling to build from multiple repos. I’m open to including this if possible. Until that tool gets open sourced, I think this is a step forward for TC builds.
https://github.com/apache/trafficcontrol/pull/2399
Any concerns with bringing this in?
—Eric
Re: PR to improve build process
Posted by Chris Lemmons <al...@gmail.com>.
Yeah, we used to have astats separate, but during upgrades, we had to
carefully keep track of which astats were compatible with which ATS.
It was easier to just combine them. We use this snippet in our build
script:
# Patch astats in so that it builds in-tree.
cp -far /opt/src/astats_over_http
/rpmbuilddir/SOURCES/src/plugins/astats_over_http
cat > /rpmbuilddir/SOURCES/src/plugins/astats_over_http/Makefile.inc <<MAKEFILE
pkglib_LTLIBRARIES += astats_over_http/astats_over_http.la
astats_over_http_astats_over_http_la_SOURCES =
astats_over_http/astats_over_http.c
MAKEFILE
(ed /rpmbuilddir/SOURCES/src/plugins/Makefile.am <<ED
/stats_over_http/
t
s/stats/astats/g
w
ED
) || die "Failed to patch plugins makefile to include astats."
It's ugly, but works.
As for a "commitment to maintaining an up to date list of patches",
well, that's the job of this community, and we've made a commitment to
be part of it. :) So, while I'm sure to keep it up-to-date, you're
just as welcome to do so. It's not an arduous task in any case. (And
when it is, it needs to be done all the more urgently.)
The stuff that winds up in the patches list is:
- New features that aren't really candidates for backporting, usually plugins.
- Very specific changes that are required for ATC (like the presence
of a spec file).
- Bug fixes that are available upstream, but not yet backported and released.
That last category only exists because ATC runs at a gallop compared
to ATS's steady saunter. Those tend to be very short lived, but it's
exceedingly handy to have them.
On Fri, Jun 15, 2018 at 2:09 PM, Eric Friedrich (efriedri)
<ef...@cisco.com.invalid> wrote:
> Thanks for comments Chris, I agree with all of them.
>
> Do you typically include astats in your traffic server tar balls? Based on whats in Github, its traditionally been shipped as a separate RPM.
>
> I agree build would be simpler with astats packaged in, especially because the TS version dependencies is some of the most complicated of my changes.
>
> If you get TSB open, is there a commitment to maintaining an up to date list of patches that should be applied for full ATC compatibility (today just URI signing/astats)?
>
> —Eric
>
>> On Jun 15, 2018, at 4:03 PM, Chris Lemmons <al...@gmail.com> wrote:
>>
>> I have so many thoughts on this topic. :)
>>
>> We use a tool for composing the public ATS build on top of our list of
>> backports (all of which are currently public, iirc). This is how we
>> use stuff like URI Signing, which isn't available in ATS 7. I'm
>> currently in the process of getting that opened up so that everyone
>> can look at it (a couple weeks, probably, though), but the principle
>> behind it isn't exceptionally complex.
>>
>> What I'd like to see in the ATC ATS RPM is:
>> - It's deterministically produced (basically, everything (as far as
>> is reasonable) pegged to specific changesets/versions).
>> - It contains every feature supported by ATC.
>> - It builds as a `./pkg` target: `./pkg ats_build`.
>> - It produces a single RPM that just works.
>> Bonus points if it's relatively easy to add extra backports or private
>> bug fixes.
>>
>> The PR is close on determinism, though it encourages using a branch
>> instead of a revision, which could cause trouble for folks with RPMs
>> updating unexpectedly. It doesn't have backports for URI Signing and
>> astats is in its own RPM, which I think puts more work on the people
>> managing the CDN than is necessary. It's spot on for the ./pkg usage.
>>
>> It's absolutely a step in the right direction, though, and I'd rather
>> have something than nothing. I'm not totally enthused with this
>> solution, but it's a start and we can work toward things going
>> forward.
>>
>> On Fri, Jun 15, 2018 at 12:48 PM, Zelkowitz, Evan
>> <Ev...@comcast.com> wrote:
>>> I like the idea and looks good. One thing to think about though is in the trafficserver.spec file, if there are any good ways to dynamically do those config files includes at the end. Im no spec expert so Im not sure if there may be a better way but recently here I worked on getting our internal spec and build system to be able to do builds of ATS master and it looks like starting with 8/9 some more of those files will be going away and another new one introduced. So those files change frequently across version numbers
>>>
>>> On 6/15/18, 11:27 AM, "Eric Friedrich (efriedri)" <ef...@cisco.com.INVALID> wrote:
>>>
>>> I’ve opened a PR to produce Trafficserver and astats RPMs in the Traffic Control build process. Previously, new users would need to create their own build environments or ask for RPMS on Slack.
>>>
>>> This PR will lower the barrier to entry for new users. For more involved users, theres now a set of scripts that will build inside a Dockerized Jenkins environment.
>>>
>>> The changes rely on having all TS changes in a single git repository to form the RPM source tarball. I understand there are some users which have private tooling to build from multiple repos. I’m open to including this if possible. Until that tool gets open sourced, I think this is a step forward for TC builds.
>>>
>>> https://github.com/apache/trafficcontrol/pull/2399
>>>
>>> Any concerns with bringing this in?
>>>
>>> —Eric
>>>
>>>
>>>
>
Re: PR to improve build process
Posted by "Eric Friedrich (efriedri)" <ef...@cisco.com.INVALID>.
Thanks for comments Chris, I agree with all of them.
Do you typically include astats in your traffic server tar balls? Based on whats in Github, its traditionally been shipped as a separate RPM.
I agree build would be simpler with astats packaged in, especially because the TS version dependencies is some of the most complicated of my changes.
If you get TSB open, is there a commitment to maintaining an up to date list of patches that should be applied for full ATC compatibility (today just URI signing/astats)?
—Eric
> On Jun 15, 2018, at 4:03 PM, Chris Lemmons <al...@gmail.com> wrote:
>
> I have so many thoughts on this topic. :)
>
> We use a tool for composing the public ATS build on top of our list of
> backports (all of which are currently public, iirc). This is how we
> use stuff like URI Signing, which isn't available in ATS 7. I'm
> currently in the process of getting that opened up so that everyone
> can look at it (a couple weeks, probably, though), but the principle
> behind it isn't exceptionally complex.
>
> What I'd like to see in the ATC ATS RPM is:
> - It's deterministically produced (basically, everything (as far as
> is reasonable) pegged to specific changesets/versions).
> - It contains every feature supported by ATC.
> - It builds as a `./pkg` target: `./pkg ats_build`.
> - It produces a single RPM that just works.
> Bonus points if it's relatively easy to add extra backports or private
> bug fixes.
>
> The PR is close on determinism, though it encourages using a branch
> instead of a revision, which could cause trouble for folks with RPMs
> updating unexpectedly. It doesn't have backports for URI Signing and
> astats is in its own RPM, which I think puts more work on the people
> managing the CDN than is necessary. It's spot on for the ./pkg usage.
>
> It's absolutely a step in the right direction, though, and I'd rather
> have something than nothing. I'm not totally enthused with this
> solution, but it's a start and we can work toward things going
> forward.
>
> On Fri, Jun 15, 2018 at 12:48 PM, Zelkowitz, Evan
> <Ev...@comcast.com> wrote:
>> I like the idea and looks good. One thing to think about though is in the trafficserver.spec file, if there are any good ways to dynamically do those config files includes at the end. Im no spec expert so Im not sure if there may be a better way but recently here I worked on getting our internal spec and build system to be able to do builds of ATS master and it looks like starting with 8/9 some more of those files will be going away and another new one introduced. So those files change frequently across version numbers
>>
>> On 6/15/18, 11:27 AM, "Eric Friedrich (efriedri)" <ef...@cisco.com.INVALID> wrote:
>>
>> I’ve opened a PR to produce Trafficserver and astats RPMs in the Traffic Control build process. Previously, new users would need to create their own build environments or ask for RPMS on Slack.
>>
>> This PR will lower the barrier to entry for new users. For more involved users, theres now a set of scripts that will build inside a Dockerized Jenkins environment.
>>
>> The changes rely on having all TS changes in a single git repository to form the RPM source tarball. I understand there are some users which have private tooling to build from multiple repos. I’m open to including this if possible. Until that tool gets open sourced, I think this is a step forward for TC builds.
>>
>> https://github.com/apache/trafficcontrol/pull/2399
>>
>> Any concerns with bringing this in?
>>
>> —Eric
>>
>>
>>
Re: PR to improve build process
Posted by Chris Lemmons <al...@gmail.com>.
I have so many thoughts on this topic. :)
We use a tool for composing the public ATS build on top of our list of
backports (all of which are currently public, iirc). This is how we
use stuff like URI Signing, which isn't available in ATS 7. I'm
currently in the process of getting that opened up so that everyone
can look at it (a couple weeks, probably, though), but the principle
behind it isn't exceptionally complex.
What I'd like to see in the ATC ATS RPM is:
- It's deterministically produced (basically, everything (as far as
is reasonable) pegged to specific changesets/versions).
- It contains every feature supported by ATC.
- It builds as a `./pkg` target: `./pkg ats_build`.
- It produces a single RPM that just works.
Bonus points if it's relatively easy to add extra backports or private
bug fixes.
The PR is close on determinism, though it encourages using a branch
instead of a revision, which could cause trouble for folks with RPMs
updating unexpectedly. It doesn't have backports for URI Signing and
astats is in its own RPM, which I think puts more work on the people
managing the CDN than is necessary. It's spot on for the ./pkg usage.
It's absolutely a step in the right direction, though, and I'd rather
have something than nothing. I'm not totally enthused with this
solution, but it's a start and we can work toward things going
forward.
On Fri, Jun 15, 2018 at 12:48 PM, Zelkowitz, Evan
<Ev...@comcast.com> wrote:
> I like the idea and looks good. One thing to think about though is in the trafficserver.spec file, if there are any good ways to dynamically do those config files includes at the end. Im no spec expert so Im not sure if there may be a better way but recently here I worked on getting our internal spec and build system to be able to do builds of ATS master and it looks like starting with 8/9 some more of those files will be going away and another new one introduced. So those files change frequently across version numbers
>
> On 6/15/18, 11:27 AM, "Eric Friedrich (efriedri)" <ef...@cisco.com.INVALID> wrote:
>
> I’ve opened a PR to produce Trafficserver and astats RPMs in the Traffic Control build process. Previously, new users would need to create their own build environments or ask for RPMS on Slack.
>
> This PR will lower the barrier to entry for new users. For more involved users, theres now a set of scripts that will build inside a Dockerized Jenkins environment.
>
> The changes rely on having all TS changes in a single git repository to form the RPM source tarball. I understand there are some users which have private tooling to build from multiple repos. I’m open to including this if possible. Until that tool gets open sourced, I think this is a step forward for TC builds.
>
> https://github.com/apache/trafficcontrol/pull/2399
>
> Any concerns with bringing this in?
>
> —Eric
>
>
>
Re: PR to improve build process
Posted by "Zelkowitz, Evan" <Ev...@comcast.com>.
I like the idea and looks good. One thing to think about though is in the trafficserver.spec file, if there are any good ways to dynamically do those config files includes at the end. Im no spec expert so Im not sure if there may be a better way but recently here I worked on getting our internal spec and build system to be able to do builds of ATS master and it looks like starting with 8/9 some more of those files will be going away and another new one introduced. So those files change frequently across version numbers
On 6/15/18, 11:27 AM, "Eric Friedrich (efriedri)" <ef...@cisco.com.INVALID> wrote:
I’ve opened a PR to produce Trafficserver and astats RPMs in the Traffic Control build process. Previously, new users would need to create their own build environments or ask for RPMS on Slack.
This PR will lower the barrier to entry for new users. For more involved users, theres now a set of scripts that will build inside a Dockerized Jenkins environment.
The changes rely on having all TS changes in a single git repository to form the RPM source tarball. I understand there are some users which have private tooling to build from multiple repos. I’m open to including this if possible. Until that tool gets open sourced, I think this is a step forward for TC builds.
https://github.com/apache/trafficcontrol/pull/2399
Any concerns with bringing this in?
—Eric