Unit test logs in Jenkins?

[Marcelo Vanzin <va...@cloudera.com>]

Hey all,

Is there a way to access unit test logs in jenkins builds? e.g.,
core/target/unit-tests.log

That would be really helpful to debug build failures. The scalatest
output isn't all that helpful.

If that's currently not available, would it be possible to add those
logs as build artifacts?

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Test all the things (Was: Unit test logs in Jenkins?)

[shane knapp <sk...@berkeley.edu>]

cool.  FYI, i'm at databricks today and talked w/patrick, josh and davies
about this.  we have some great ideas to actually make this happen and will
be pushing over the next few weeks to get it done.  :)

On Thu, Apr 2, 2015 at 9:21 AM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> (Renaming thread so as to un-hijack Marcelo's request.)
>
> Sure, we definitely want tests running faster.
>
> Part of "testing all the things" will be factoring out stuff from the
> various builds that can be run just once.
>
> We've also tried in the past (with little success) to parallelize test
> execution <https://issues.apache.org/jira/browse/SPARK-3431>. That still
> needs work before it becomes possible.
>
> Nick
>
>
> On Thu, Apr 2, 2015 at 11:59 AM shane knapp <sk...@berkeley.edu> wrote:
>
>> i agree with all of this.  but can we please break up the tests and make
>> them shorter?  :)
>>
>> On Thu, Apr 2, 2015 at 8:54 AM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> This is secondary to Marcelo’s question, but I wanted to comment on this:
>>>
>>> Its main limitation is more cultural than technical: you need to get
>>> people
>>> to care about intermittent test runs, otherwise you can end up with
>>> failures that nobody keeps on top of
>>>
>>> This is a problem that plagues Spark as well, but there *is* a technical
>>> solution.
>>>
>>> The solution is simple: *All* the builds that we care about run for
>>> *every*
>>> proposed change. If *any* build fails, the change doesn’t make it into
>>> the
>>
>>
>>> repository.
>>>
>>> Spark already has a pull request builder that tests and reports back on
>>> PRs. Committers don’t merge in PRs when this builder reports that it
>>> failed
>>> some tests. That’s a good thing.
>>>
>>> The problem is that there are several other builds that we run on a fixed
>>> interval, independent of the pull request builder. These builds test
>>> different configurations, dependency versions, and environments than what
>>> the PR builder covers. If one of those builds fails, it fails on its own
>>> little island, with no-one to hear it scream. The build failure is
>>> detached
>>> from the PR that caused it to fail.
>>>
>>> What should happen is that the whole matrix of stuff we care to test gets
>>> run for every PR. No PR goes in if any build we care about fails for that
>>> PR, and every build we care about runs for every commit of every PR.
>>>
>>> Really, this is just an extension of the basic idea of the PR builder. It
>>>
>> doesn’t make much sense to test stuff *after* it has been committed and
>>
>>
>>> potentially broken things. And it becomes exponentially more difficult to
>>> find and fix a problem the longer it has been festering in the repo. It’s
>>> best to keep such problems out in the first place.
>>>
>>> With some more work on our CI infrastructure, I think this can be done.
>>> Maybe even later this year.
>>>
>>> Nick
>>>
>>
>>> On Thu, Apr 2, 2015 at 6:02 AM Steve Loughran stevel@hortonworks.com
>>>
>> <ht...@hortonworks.com> wrote:
>>>
>>>
>>> > > On 2 Apr 2015, at 06:31, Patrick Wendell <pw...@gmail.com> wrote:
>>> > >
>>> > > Hey Marcelo,
>>> > >
>>> > > Great question. Right now, some of the more active developers have an
>>> > > account that allows them to log into this cluster to inspect logs (we
>>> > > copy the logs from each run to a node on that cluster). The
>>> > > infrastructure is maintained by the AMPLab.
>>> > >
>>> > > I will put you in touch the someone there who can get you an account.
>>> > >
>>> > > This is a short term solution. The longer term solution is to have
>>> > > these scp'd regularly to an S3 bucket or somewhere people can get
>>> > > access to them, but that's not ready yet.
>>> > >
>>> > > - Patrick
>>> > >
>>> > >>
>>> >
>>> >
>>> > ASF Jenkins is always there to play with; committers/PMC members should
>>> > just need to file a BUILD JIRA to get access.
>>> >
>>> > Its main limitation is more cultural than technical: you need to get
>>> > people to care about intermittent test runs, otherwise you can end up
>>> with
>>> > failures that nobody keeps on top of
>>> > https://builds.apache.org/view/H-L/view/Hadoop/
>>> >
>>> > Someone really needs to own the "keep the builds working" problem -and
>>> > have the ability to somehow kick others into fixing things. The latter
>>> is
>>> > pretty hard cross-organisation
>>> >
>>> >
>>> > >> That would be really helpful to debug build failures. The scalatest
>>> > >> output isn't all that helpful.
>>> > >>
>>> >
>>> > Potentially an issue with the test runner, rather than the tests
>>> > themselves.
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> > For additional commands, e-mail: dev-help@spark.apache.org
>>> >
>>> >  ​
>>>
>>

Test all the things (Was: Unit test logs in Jenkins?)

[Nicholas Chammas <ni...@gmail.com>]

(Renaming thread so as to un-hijack Marcelo's request.)

Sure, we definitely want tests running faster.

Part of "testing all the things" will be factoring out stuff from the
various builds that can be run just once.

We've also tried in the past (with little success) to parallelize test
execution <https://issues.apache.org/jira/browse/SPARK-3431>. That still
needs work before it becomes possible.

Nick


On Thu, Apr 2, 2015 at 11:59 AM shane knapp <sk...@berkeley.edu> wrote:

> i agree with all of this.  but can we please break up the tests and make
> them shorter?  :)
>
> On Thu, Apr 2, 2015 at 8:54 AM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> This is secondary to Marcelo’s question, but I wanted to comment on this:
>>
>> Its main limitation is more cultural than technical: you need to get
>> people
>> to care about intermittent test runs, otherwise you can end up with
>> failures that nobody keeps on top of
>>
>> This is a problem that plagues Spark as well, but there *is* a technical
>> solution.
>>
>> The solution is simple: *All* the builds that we care about run for
>> *every*
>> proposed change. If *any* build fails, the change doesn’t make it into the
>
>
>> repository.
>>
>> Spark already has a pull request builder that tests and reports back on
>> PRs. Committers don’t merge in PRs when this builder reports that it
>> failed
>> some tests. That’s a good thing.
>>
>> The problem is that there are several other builds that we run on a fixed
>> interval, independent of the pull request builder. These builds test
>> different configurations, dependency versions, and environments than what
>> the PR builder covers. If one of those builds fails, it fails on its own
>> little island, with no-one to hear it scream. The build failure is
>> detached
>> from the PR that caused it to fail.
>>
>> What should happen is that the whole matrix of stuff we care to test gets
>> run for every PR. No PR goes in if any build we care about fails for that
>> PR, and every build we care about runs for every commit of every PR.
>>
>> Really, this is just an extension of the basic idea of the PR builder. It
>>
> doesn’t make much sense to test stuff *after* it has been committed and
>
>
>> potentially broken things. And it becomes exponentially more difficult to
>> find and fix a problem the longer it has been festering in the repo. It’s
>> best to keep such problems out in the first place.
>>
>> With some more work on our CI infrastructure, I think this can be done.
>> Maybe even later this year.
>>
>> Nick
>>
>
>> On Thu, Apr 2, 2015 at 6:02 AM Steve Loughran stevel@hortonworks.com
>>
> <ht...@hortonworks.com> wrote:
>>
>>
>> > > On 2 Apr 2015, at 06:31, Patrick Wendell <pw...@gmail.com> wrote:
>> > >
>> > > Hey Marcelo,
>> > >
>> > > Great question. Right now, some of the more active developers have an
>> > > account that allows them to log into this cluster to inspect logs (we
>> > > copy the logs from each run to a node on that cluster). The
>> > > infrastructure is maintained by the AMPLab.
>> > >
>> > > I will put you in touch the someone there who can get you an account.
>> > >
>> > > This is a short term solution. The longer term solution is to have
>> > > these scp'd regularly to an S3 bucket or somewhere people can get
>> > > access to them, but that's not ready yet.
>> > >
>> > > - Patrick
>> > >
>> > >>
>> >
>> >
>> > ASF Jenkins is always there to play with; committers/PMC members should
>> > just need to file a BUILD JIRA to get access.
>> >
>> > Its main limitation is more cultural than technical: you need to get
>> > people to care about intermittent test runs, otherwise you can end up
>> with
>> > failures that nobody keeps on top of
>> > https://builds.apache.org/view/H-L/view/Hadoop/
>> >
>> > Someone really needs to own the "keep the builds working" problem -and
>> > have the ability to somehow kick others into fixing things. The latter
>> is
>> > pretty hard cross-organisation
>> >
>> >
>> > >> That would be really helpful to debug build failures. The scalatest
>> > >> output isn't all that helpful.
>> > >>
>> >
>> > Potentially an issue with the test runner, rather than the tests
>> > themselves.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: dev-help@spark.apache.org
>> >
>> >  ​
>>
>

Re: Unit test logs in Jenkins?

[shane knapp <sk...@berkeley.edu>]

i agree with all of this.  but can we please break up the tests and make
them shorter?  :)

On Thu, Apr 2, 2015 at 8:54 AM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> This is secondary to Marcelo’s question, but I wanted to comment on this:
>
> Its main limitation is more cultural than technical: you need to get people
> to care about intermittent test runs, otherwise you can end up with
> failures that nobody keeps on top of
>
> This is a problem that plagues Spark as well, but there *is* a technical
> solution.
>
> The solution is simple: *All* the builds that we care about run for *every*
> proposed change. If *any* build fails, the change doesn’t make it into the
> repository.
>
> Spark already has a pull request builder that tests and reports back on
> PRs. Committers don’t merge in PRs when this builder reports that it failed
> some tests. That’s a good thing.
>
> The problem is that there are several other builds that we run on a fixed
> interval, independent of the pull request builder. These builds test
> different configurations, dependency versions, and environments than what
> the PR builder covers. If one of those builds fails, it fails on its own
> little island, with no-one to hear it scream. The build failure is detached
> from the PR that caused it to fail.
>
> What should happen is that the whole matrix of stuff we care to test gets
> run for every PR. No PR goes in if any build we care about fails for that
> PR, and every build we care about runs for every commit of every PR.
>
> Really, this is just an extension of the basic idea of the PR builder. It
> doesn’t make much sense to test stuff *after* it has been committed and
> potentially broken things. And it becomes exponentially more difficult to
> find and fix a problem the longer it has been festering in the repo. It’s
> best to keep such problems out in the first place.
>
> With some more work on our CI infrastructure, I think this can be done.
> Maybe even later this year.
>
> Nick
>
> On Thu, Apr 2, 2015 at 6:02 AM Steve Loughran stevel@hortonworks.com
> <ht...@hortonworks.com> wrote:
>
>
> > > On 2 Apr 2015, at 06:31, Patrick Wendell <pw...@gmail.com> wrote:
> > >
> > > Hey Marcelo,
> > >
> > > Great question. Right now, some of the more active developers have an
> > > account that allows them to log into this cluster to inspect logs (we
> > > copy the logs from each run to a node on that cluster). The
> > > infrastructure is maintained by the AMPLab.
> > >
> > > I will put you in touch the someone there who can get you an account.
> > >
> > > This is a short term solution. The longer term solution is to have
> > > these scp'd regularly to an S3 bucket or somewhere people can get
> > > access to them, but that's not ready yet.
> > >
> > > - Patrick
> > >
> > >>
> >
> >
> > ASF Jenkins is always there to play with; committers/PMC members should
> > just need to file a BUILD JIRA to get access.
> >
> > Its main limitation is more cultural than technical: you need to get
> > people to care about intermittent test runs, otherwise you can end up
> with
> > failures that nobody keeps on top of
> > https://builds.apache.org/view/H-L/view/Hadoop/
> >
> > Someone really needs to own the "keep the builds working" problem -and
> > have the ability to somehow kick others into fixing things. The latter is
> > pretty hard cross-organisation
> >
> >
> > >> That would be really helpful to debug build failures. The scalatest
> > >> output isn't all that helpful.
> > >>
> >
> > Potentially an issue with the test runner, rather than the tests
> > themselves.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
> >  ​
>

Re: Unit test logs in Jenkins?

[Nicholas Chammas <ni...@gmail.com>]

This is secondary to Marcelo’s question, but I wanted to comment on this:

Its main limitation is more cultural than technical: you need to get people
to care about intermittent test runs, otherwise you can end up with
failures that nobody keeps on top of

This is a problem that plagues Spark as well, but there *is* a technical
solution.

The solution is simple: *All* the builds that we care about run for *every*
proposed change. If *any* build fails, the change doesn’t make it into the
repository.

Spark already has a pull request builder that tests and reports back on
PRs. Committers don’t merge in PRs when this builder reports that it failed
some tests. That’s a good thing.

The problem is that there are several other builds that we run on a fixed
interval, independent of the pull request builder. These builds test
different configurations, dependency versions, and environments than what
the PR builder covers. If one of those builds fails, it fails on its own
little island, with no-one to hear it scream. The build failure is detached
from the PR that caused it to fail.

What should happen is that the whole matrix of stuff we care to test gets
run for every PR. No PR goes in if any build we care about fails for that
PR, and every build we care about runs for every commit of every PR.

Really, this is just an extension of the basic idea of the PR builder. It
doesn’t make much sense to test stuff *after* it has been committed and
potentially broken things. And it becomes exponentially more difficult to
find and fix a problem the longer it has been festering in the repo. It’s
best to keep such problems out in the first place.

With some more work on our CI infrastructure, I think this can be done.
Maybe even later this year.

Nick

On Thu, Apr 2, 2015 at 6:02 AM Steve Loughran stevel@hortonworks.com
<ht...@hortonworks.com> wrote:


> > On 2 Apr 2015, at 06:31, Patrick Wendell <pw...@gmail.com> wrote:
> >
> > Hey Marcelo,
> >
> > Great question. Right now, some of the more active developers have an
> > account that allows them to log into this cluster to inspect logs (we
> > copy the logs from each run to a node on that cluster). The
> > infrastructure is maintained by the AMPLab.
> >
> > I will put you in touch the someone there who can get you an account.
> >
> > This is a short term solution. The longer term solution is to have
> > these scp'd regularly to an S3 bucket or somewhere people can get
> > access to them, but that's not ready yet.
> >
> > - Patrick
> >
> >>
>
>
> ASF Jenkins is always there to play with; committers/PMC members should
> just need to file a BUILD JIRA to get access.
>
> Its main limitation is more cultural than technical: you need to get
> people to care about intermittent test runs, otherwise you can end up with
> failures that nobody keeps on top of
> https://builds.apache.org/view/H-L/view/Hadoop/
>
> Someone really needs to own the "keep the builds working" problem -and
> have the ability to somehow kick others into fixing things. The latter is
> pretty hard cross-organisation
>
>
> >> That would be really helpful to debug build failures. The scalatest
> >> output isn't all that helpful.
> >>
>
> Potentially an issue with the test runner, rather than the tests
> themselves.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>  ​

Re: Unit test logs in Jenkins?

[Steve Loughran <st...@hortonworks.com>]

> On 2 Apr 2015, at 19:44, Marcelo Vanzin <va...@cloudera.com> wrote:
> 
> On Thu, Apr 2, 2015 at 3:01 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>>> That would be really helpful to debug build failures. The scalatest
>>>> output isn't all that helpful.
>>>> 
>> 
>> Potentially an issue with the test runner, rather than the tests themselves.
> 
> Sorry, that was me over-generalizing. The output is generally fine,
> except for certain classes of tests (especially those that need to
> fork a child process or even multiple processes). In those cases, most
> of the interesting output ends up in logs, and the error reported by
> scalatest is very generic.
> 
> -- 
> Marcelo


I've been seeing (and remember, I've only recently started playing with funsuite), that the logs go to target/unit.log; edit log4j.properties to go to stdout and they go to stdout -but they don't go into the surefire logs. That is, the test runner isn't generating the classic <junit> XML report with its sections of stdout and stderr. While that format has its limitations, and you can certainly do better (ideally every log level event should be captured; let you turn them on/off, pull in threads, correlate timestamps across machines, etc, etc), having Junit & jenkins reports which include all the output logs testcase-by-testcase is pretty foundational. Maybe there's some way to turn it on that I haven't seen. 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Unit test logs in Jenkins?

[Marcelo Vanzin <va...@cloudera.com>]

On Thu, Apr 2, 2015 at 3:01 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>> That would be really helpful to debug build failures. The scalatest
>>> output isn't all that helpful.
>>>
>
> Potentially an issue with the test runner, rather than the tests themselves.

Sorry, that was me over-generalizing. The output is generally fine,
except for certain classes of tests (especially those that need to
fork a child process or even multiple processes). In those cases, most
of the interesting output ends up in logs, and the error reported by
scalatest is very generic.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Unit test logs in Jenkins?

[Steve Loughran <st...@hortonworks.com>]

> On 2 Apr 2015, at 06:31, Patrick Wendell <pw...@gmail.com> wrote:
> 
> Hey Marcelo,
> 
> Great question. Right now, some of the more active developers have an
> account that allows them to log into this cluster to inspect logs (we
> copy the logs from each run to a node on that cluster). The
> infrastructure is maintained by the AMPLab.
> 
> I will put you in touch the someone there who can get you an account.
> 
> This is a short term solution. The longer term solution is to have
> these scp'd regularly to an S3 bucket or somewhere people can get
> access to them, but that's not ready yet.
> 
> - Patrick
> 
>> 


ASF Jenkins is always there to play with; committers/PMC members should just need to file a BUILD JIRA to get access.

Its main limitation is more cultural than technical: you need to get people to care about intermittent test runs, otherwise you can end up with failures that nobody keeps on top of
https://builds.apache.org/view/H-L/view/Hadoop/

Someone really needs to own the "keep the builds working" problem -and have the ability to somehow kick others into fixing things. The latter is pretty hard cross-organisation


>> That would be really helpful to debug build failures. The scalatest
>> output isn't all that helpful.
>> 

Potentially an issue with the test runner, rather than the tests themselves.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: Unit test logs in Jenkins?

[Patrick Wendell <pw...@gmail.com>]

Hey Marcelo,

Great question. Right now, some of the more active developers have an
account that allows them to log into this cluster to inspect logs (we
copy the logs from each run to a node on that cluster). The
infrastructure is maintained by the AMPLab.

I will put you in touch the someone there who can get you an account.

This is a short term solution. The longer term solution is to have
these scp'd regularly to an S3 bucket or somewhere people can get
access to them, but that's not ready yet.

- Patrick



On Wed, Apr 1, 2015 at 1:01 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Hey all,
>
> Is there a way to access unit test logs in jenkins builds? e.g.,
> core/target/unit-tests.log
>
> That would be really helpful to debug build failures. The scalatest
> output isn't all that helpful.
>
> If that's currently not available, would it be possible to add those
> logs as build artifacts?
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org