You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael Wechner <mi...@wyona.com> on 2014/12/02 23:31:10 UTC

Running tests incrementally based on changed files

Hi

I am trying to setup a testing environment, which is running tests
incrementally based on changed files, instead of running whole builds.

I would like to try this for Lucene/Solr, hence I am trying to
understand better the relationship between individual code changes and
associated tests. For example I have noticed to the following change:

git diff --name-only 641ed06..cb4fc89
lucene/core/src/java/org/apache/lucene/codecs/compressing/CompressingStoredFieldsFormat.java

and if I understand correctly, then the following tests would have to be executed:

lucene/core/src/test/org/apache/lucene/codecs/compressing/TestCompressingStoredFieldsFormat.java
lucene/test-framework/src/java/org/apache/lucene/codecs/compressing/CompressingCodec.java

is that correct?

I have also noticed the following changes

git diff --name-only 641ed06..cb4fc89
solr/cloud-dev/clean.sh
solr/cloud-dev/cli-test-solrcloud-start.sh
solr/cloud-dev/functions.sh
solr/cloud-dev/solrcloud-extzk-start.sh
solr/cloud-dev/solrcloud-multi-start.sh
solr/cloud-dev/solrcloud-start-existing.sh
solr/cloud-dev/solrcloud-start.sh
solr/cloud-dev/stop.sh

I guess there are no automatic tests for these changes, but somebody would have to manually test the changes of these
scripts, right?


Thanks in advance for helping me to try to better understand this.

Michael




Re: Running tests incrementally based on changed files

Posted by Michael Wechner <mi...@wyona.com>.
Am 03.12.14 um 07:26 schrieb Shawn Heisey:
> On 12/2/2014 10:47 PM, Michael Wechner wrote:
>> I think you are right that in certain cases, like for example in the
>> case of CompressingStoredFieldsFormat
>> it makes a lot of sense to run the entire test suit (even multiple times
>> considering randomization), but I would argue that there are probably
>> quite a few other cases where incremental testing would be sufficient,
>> at least as a first shot.
> I do like to run the entire test suite at least once before I commit a
> change.  Usually I don't work on pieces of the system that are quite
> this central ... At my current skill level, I am more comfortable at the
> edges, where the code that I write is not expected to hold up huge
> sections of the entire project.
>
> Knowing what tests to run during development of a particular bugfix or
> change (when you definitely don't want to wait for the whole test suite)
> is something that comes with a lot of experience.

right and I think exactly this experience should be documented and I think
it can be documented in a very simple manner, which will help a lot and
the cost for maintaining this information is way less than what you will gain.




>   I will often resort
> to asking dumb questions in Jira or on this list.  Writing a new test
> for the specific feature or problem you're working on is something we
> definitely encourage.

right and this is the other point I would like to make, that with every
change, the testing system
would be able to determine which tests to run and if there are no tests
or not enough tests yet, then it
will encourage/remind the developer in a friendly manner to write one.
In german one would say

"steter Tropfen höhlt den Stein"

I guess it translates to

"constant dropping wears away a stone "




>   Later review might find that it's redundant and
> one of them might get removed, but a lengthy test suite is a small price
> to pay for code that can be trusted.

I am really not against a lengthy test suite, it is really not one
against the other. It's about what can be done to further improve the testing,
also with regard to agile development, using bug and feature branches for "small" changes.


>
> With IDE tools like Eclipse or Idea, you can trace the derivation and
> usage of each class, which can give you hints about which tests might be
> appropriate, but the interconnections can quickly become too complicated
> to track.

I think that's where "human" experience can help, especially when it is documented and this
documentation is being maintained on a regular basis, which I think I have found a way to get done.

I guess it is best if I will create some concrete examples and then it might be clearer
what I am trying to achieve.

Thanks very much for all your feedback so far

Michael




>
> Every time I go diving into the source code, I am impressed and daunted
> by the complex web of inheritance and interdependence that Lucene/Solr
> contains.  From what I've seen, our codebase makes effective use of just
> about every capability that Java offers.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Michael Wechner <mi...@wyona.com>.
Am 03.12.14 um 07:26 schrieb Shawn Heisey:
> On 12/2/2014 10:47 PM, Michael Wechner wrote:
>> I think you are right that in certain cases, like for example in the
>> case of CompressingStoredFieldsFormat
>> it makes a lot of sense to run the entire test suit (even multiple times
>> considering randomization), but I would argue that there are probably
>> quite a few other cases where incremental testing would be sufficient,
>> at least as a first shot.
> I do like to run the entire test suite at least once before I commit a
> change.  Usually I don't work on pieces of the system that are quite
> this central ... At my current skill level, I am more comfortable at the
> edges, where the code that I write is not expected to hold up huge
> sections of the entire project.
>
> Knowing what tests to run during development of a particular bugfix or
> change (when you definitely don't want to wait for the whole test suite)
> is something that comes with a lot of experience. 

right and I think exactly this experience should be documented and I think
it can be documented in a very simple manner, which will help a lot and
the cost
for maintaining this information is way less what you will gain.
>  I will often resort
> to asking dumb questions in Jira or on this list.  Writing a new test
> for the specific feature or problem you're working on is something we
> definitely encourage. 

right and this is the other point I would like to make, that with every
change, the testing system
would be able to determine which tests to run and if there are no tests
or not enough tests yet, then it
will encourage/remind the developer in a friendly manner to write one.
In german one would say

"steter Tropfen höhlt den Stein"

I guess it translates to

"constant dropping wears away a stone "

:-)

>  Later review might find that it's redundant and
> one of them might get removed, but a lengthy test suite is a small price
> to pay for code that can be trusted.

I am really not against a lengthy test suite, it is really not one
against the other.
>
> With IDE tools like Eclipse or Idea, you can trace the derivation and
> usage of each class, which can give you hints about which tests might be
> appropriate, but the interconnections can quickly become too complicated
> to track.

I think that's where "human" experience can help

Thanks

Michael
>
> Every time I go diving into the source code, I am impressed and daunted
> by the complex web of inheritance and interdependence that Lucene/Solr
> contains.  From what I've seen, our codebase makes effective use of just
> about every capability that Java offers.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>


Re: Running tests incrementally based on changed files

Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/2/2014 10:47 PM, Michael Wechner wrote:
> I think you are right that in certain cases, like for example in the
> case of CompressingStoredFieldsFormat
> it makes a lot of sense to run the entire test suit (even multiple times
> considering randomization), but I would argue that there are probably
> quite a few other cases where incremental testing would be sufficient,
> at least as a first shot.

I do like to run the entire test suite at least once before I commit a
change.  Usually I don't work on pieces of the system that are quite
this central ... At my current skill level, I am more comfortable at the
edges, where the code that I write is not expected to hold up huge
sections of the entire project.

Knowing what tests to run during development of a particular bugfix or
change (when you definitely don't want to wait for the whole test suite)
is something that comes with a lot of experience.  I will often resort
to asking dumb questions in Jira or on this list.  Writing a new test
for the specific feature or problem you're working on is something we
definitely encourage.  Later review might find that it's redundant and
one of them might get removed, but a lengthy test suite is a small price
to pay for code that can be trusted.

With IDE tools like Eclipse or Idea, you can trace the derivation and
usage of each class, which can give you hints about which tests might be
appropriate, but the interconnections can quickly become too complicated
to track.

Every time I go diving into the source code, I am impressed and daunted
by the complex web of inheritance and interdependence that Lucene/Solr
contains.  From what I've seen, our codebase makes effective use of just
about every capability that Java offers.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Michael Wechner <mi...@wyona.com>.
Am 03.12.14 um 06:53 schrieb Alexandre Rafalovitch:
> A tangent, but a relevant one (to the issue of speed). Have you tried
> running the tests with Lucene/Solr code being in the RAM disk?

no, but that's a good idea re speed.


>
> I found that compiling source on RAMDisk is a lot faster than even
> with SSD drive. Must be just frequency of access.

how much faster is it approximately?

Thanks

Michael

>  It might be the same
> with tests.
>
> Regards,
>    Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 3 December 2014 at 00:47, Michael Wechner <mi...@wyona.com> wrote:
>> Hi Shawn
>>
>> Thanks very much for your feedback. Please see my comments inline below
>>
>> Am 03.12.14 um 00:10 schrieb Shawn Heisey:
>>> On 12/2/2014 3:31 PM, Michael Wechner wrote:
>>>> I am trying to setup a testing environment, which is running tests
>>>> incrementally based on changed files, instead of running whole builds.
>>>>
>>>> I would like to try this for Lucene/Solr, hence I am trying to
>>>> understand better the relationship between individual code changes and
>>>> associated tests. For example I have noticed to the following change:
>>>>
>>>> git diff --name-only 641ed06..cb4fc89
>>>> lucene/core/src/java/org/apache/lucene/codecs/compressing/CompressingStoredFieldsFormat.java
>>>>
>>>> and if I understand correctly, then the following tests would have to be executed:
>>>>
>>>> lucene/core/src/test/org/apache/lucene/codecs/compressing/TestCompressingStoredFieldsFormat.java
>>>> lucene/test-framework/src/java/org/apache/lucene/codecs/compressing/CompressingCodec.java
>>>>
>>>> is that correct?
>>> The test dependency tree for this particular class is *EXTENSIVE* and
>>> not very easy to track down.
>>>
>>> CompressingStoredFieldsFormat is the parent class for
>>> Lucene50StoredFieldsFormat.  It is directly referenced by three classes
>>> in addition to the direct descendant.  Here's a screenshot showing all
>>> direct references in branch_5x:
>>>
>>> https://www.dropbox.com/s/3pb4kljvdbd6xug/CompressingStoredFieldsFormat-eclipse.png?dl=0
>>>
>>> Lucene50StoredFieldsFormat is used by a few classes, one of which is
>>> Lucene50Codec:
>>>
>>> https://www.dropbox.com/s/0ibnw87m6rveinv/Lucene50StoredFieldsFormat.png?dl=0
>>>
>>> Lucene50Codec is extremely central to almost everything the unreleased
>>> Lucene 5.x does if it is using default settings, so a very large number
>>> of tests would be affected by a change in the compressed stored field
>>> format.  Just for giggles, I grabbed a screenshot of the classes that
>>> directly reference Lucene50Codec:
>>>
>>> https://www.dropbox.com/s/eqq30uvckc8b19d/Lucene50Codec.png?dl=0
>> thank you very much for your insights. I did not realize that this
>> particular case has
>> such extensive dependencies
>>> The build system includes the ability to use Clover if it is present ...
>>> I don't know much about it, except that it's related to determining test
>>> coverage for parts of the software.  Perhaps a clover report could give
>>> you the info you need?
>>>
>>> https://www.atlassian.com/software/clover/overview
>> I have heard about Clover, but never used it. I will have a closer look
>> at it now
>>> Aside from that ... running the entire test suite multiple times
>>> whenever you change anything is the best option.  The test system
>>> randomizes a large number of options on each test run, so running the
>>> tests multiple times will test many different combinations.  A very
>>> large number of bugs have been discovered and fixed because of that
>>> randomization.
>>
>> I think you are right that in certain cases, like for example in the
>> case of CompressingStoredFieldsFormat
>> it makes a lot of sense to run the entire test suit (even multiple times
>> considering randomization), but I would argue that there are probably
>> quite a few other cases where incremental testing would be sufficient,
>> at least as a first shot.
>>
>> I am saying this, because I think for larger projects running entire
>> test suits do not scale well and that it is rather unefficient and
>> inconvenient for developers to run the entire test suite every time,
>> before they commit/push a tiny change.
>>
>> Since I am not familiar enough with the code of Lucene/Solr I cannot say
>> whether all code is covered by tests,
>> but I would assume it is not. Also it would be interesting to know
>> whether developers do manual testing before they commit/push or whether
>> they just rely on running the entire test suit. I guess they are doing
>> manual tests as well, but probably only test things which they think are
>> related to the changes which they made.
>>
>> Hence I think it would help developers to have a system which supports
>> incremental testing. I really do not mean to say that it should replace
>> the entire suite testing approach, but it would be additionally.
>>
>> I will think about it some more :-)
>>
>> Thanks
>>
>> Michael
>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Hmm. May have to retract that. Just tried running "ant test" on SSD
and on RAM and saw no difference worth mentioning (nearly 1 hour
each!). So maybe not useful for that.

I _thought_ it was faster for compilation, which is mostly file access
and no waiting for cloud instances to spin up. I'll have to retest
that later.

I did this on Mac where I have the following script:
# Make 1Gb RAM Disk (1024*2048), where 2048 is block count in a megabyte
diskutil erasevolume HFS+ 'RAMDisk' `hdiutil attach -nomount ram://2097152`

Though 1Gb was not enough for the "ant test" and I had to recreate the
disk with 2G. On an 8G total Mac Book Pro.

On Linux I think it requires a memory-backed loop device filesystem.
Or some such.

Regards,
    Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 3 December 2014 at 01:14, Shawn Heisey <ap...@elyograg.org> wrote:
> On 12/2/2014 10:53 PM, Alexandre Rafalovitch wrote:
>> A tangent, but a relevant one (to the issue of speed). Have you tried
>> running the tests with Lucene/Solr code being in the RAM disk?
>>
>> I found that compiling source on RAMDisk is a lot faster than even
>> with SSD drive. Must be just frequency of access. It might be the same
>> with tests.
>
> I am intrigued by this.  Do you have a quick and dirty guide to doing
> this on Linux?  How much RAM must be allocated for a typical build and
> test run?  Probably wouldn't want to run Monster tests on it. :)
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/2/2014 10:53 PM, Alexandre Rafalovitch wrote:
> A tangent, but a relevant one (to the issue of speed). Have you tried
> running the tests with Lucene/Solr code being in the RAM disk?
> 
> I found that compiling source on RAMDisk is a lot faster than even
> with SSD drive. Must be just frequency of access. It might be the same
> with tests.

I am intrigued by this.  Do you have a quick and dirty guide to doing
this on Linux?  How much RAM must be allocated for a typical build and
test run?  Probably wouldn't want to run Monster tests on it. :)

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
A tangent, but a relevant one (to the issue of speed). Have you tried
running the tests with Lucene/Solr code being in the RAM disk?

I found that compiling source on RAMDisk is a lot faster than even
with SSD drive. Must be just frequency of access. It might be the same
with tests.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 3 December 2014 at 00:47, Michael Wechner <mi...@wyona.com> wrote:
> Hi Shawn
>
> Thanks very much for your feedback. Please see my comments inline below
>
> Am 03.12.14 um 00:10 schrieb Shawn Heisey:
>> On 12/2/2014 3:31 PM, Michael Wechner wrote:
>>> I am trying to setup a testing environment, which is running tests
>>> incrementally based on changed files, instead of running whole builds.
>>>
>>> I would like to try this for Lucene/Solr, hence I am trying to
>>> understand better the relationship between individual code changes and
>>> associated tests. For example I have noticed to the following change:
>>>
>>> git diff --name-only 641ed06..cb4fc89
>>> lucene/core/src/java/org/apache/lucene/codecs/compressing/CompressingStoredFieldsFormat.java
>>>
>>> and if I understand correctly, then the following tests would have to be executed:
>>>
>>> lucene/core/src/test/org/apache/lucene/codecs/compressing/TestCompressingStoredFieldsFormat.java
>>> lucene/test-framework/src/java/org/apache/lucene/codecs/compressing/CompressingCodec.java
>>>
>>> is that correct?
>> The test dependency tree for this particular class is *EXTENSIVE* and
>> not very easy to track down.
>>
>> CompressingStoredFieldsFormat is the parent class for
>> Lucene50StoredFieldsFormat.  It is directly referenced by three classes
>> in addition to the direct descendant.  Here's a screenshot showing all
>> direct references in branch_5x:
>>
>> https://www.dropbox.com/s/3pb4kljvdbd6xug/CompressingStoredFieldsFormat-eclipse.png?dl=0
>>
>> Lucene50StoredFieldsFormat is used by a few classes, one of which is
>> Lucene50Codec:
>>
>> https://www.dropbox.com/s/0ibnw87m6rveinv/Lucene50StoredFieldsFormat.png?dl=0
>>
>> Lucene50Codec is extremely central to almost everything the unreleased
>> Lucene 5.x does if it is using default settings, so a very large number
>> of tests would be affected by a change in the compressed stored field
>> format.  Just for giggles, I grabbed a screenshot of the classes that
>> directly reference Lucene50Codec:
>>
>> https://www.dropbox.com/s/eqq30uvckc8b19d/Lucene50Codec.png?dl=0
>
> thank you very much for your insights. I did not realize that this
> particular case has
> such extensive dependencies
>>
>> The build system includes the ability to use Clover if it is present ...
>> I don't know much about it, except that it's related to determining test
>> coverage for parts of the software.  Perhaps a clover report could give
>> you the info you need?
>>
>> https://www.atlassian.com/software/clover/overview
>
> I have heard about Clover, but never used it. I will have a closer look
> at it now
>>
>> Aside from that ... running the entire test suite multiple times
>> whenever you change anything is the best option.  The test system
>> randomizes a large number of options on each test run, so running the
>> tests multiple times will test many different combinations.  A very
>> large number of bugs have been discovered and fixed because of that
>> randomization.
>
>
> I think you are right that in certain cases, like for example in the
> case of CompressingStoredFieldsFormat
> it makes a lot of sense to run the entire test suit (even multiple times
> considering randomization), but I would argue that there are probably
> quite a few other cases where incremental testing would be sufficient,
> at least as a first shot.
>
> I am saying this, because I think for larger projects running entire
> test suits do not scale well and that it is rather unefficient and
> inconvenient for developers to run the entire test suite every time,
> before they commit/push a tiny change.
>
> Since I am not familiar enough with the code of Lucene/Solr I cannot say
> whether all code is covered by tests,
> but I would assume it is not. Also it would be interesting to know
> whether developers do manual testing before they commit/push or whether
> they just rely on running the entire test suit. I guess they are doing
> manual tests as well, but probably only test things which they think are
> related to the changes which they made.
>
> Hence I think it would help developers to have a system which supports
> incremental testing. I really do not mean to say that it should replace
> the entire suite testing approach, but it would be additionally.
>
> I will think about it some more :-)
>
> Thanks
>
> Michael
>
>>
>> Thanks,
>> Shawn
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Michael Wechner <mi...@wyona.com>.
Hi Shawn

Thanks very much for your feedback. Please see my comments inline below

Am 03.12.14 um 00:10 schrieb Shawn Heisey:
> On 12/2/2014 3:31 PM, Michael Wechner wrote:
>> I am trying to setup a testing environment, which is running tests
>> incrementally based on changed files, instead of running whole builds.
>>
>> I would like to try this for Lucene/Solr, hence I am trying to
>> understand better the relationship between individual code changes and
>> associated tests. For example I have noticed to the following change:
>>
>> git diff --name-only 641ed06..cb4fc89
>> lucene/core/src/java/org/apache/lucene/codecs/compressing/CompressingStoredFieldsFormat.java
>>
>> and if I understand correctly, then the following tests would have to be executed:
>>
>> lucene/core/src/test/org/apache/lucene/codecs/compressing/TestCompressingStoredFieldsFormat.java
>> lucene/test-framework/src/java/org/apache/lucene/codecs/compressing/CompressingCodec.java
>>
>> is that correct?
> The test dependency tree for this particular class is *EXTENSIVE* and
> not very easy to track down.
>
> CompressingStoredFieldsFormat is the parent class for
> Lucene50StoredFieldsFormat.  It is directly referenced by three classes
> in addition to the direct descendant.  Here's a screenshot showing all
> direct references in branch_5x:
>
> https://www.dropbox.com/s/3pb4kljvdbd6xug/CompressingStoredFieldsFormat-eclipse.png?dl=0
>
> Lucene50StoredFieldsFormat is used by a few classes, one of which is
> Lucene50Codec:
>
> https://www.dropbox.com/s/0ibnw87m6rveinv/Lucene50StoredFieldsFormat.png?dl=0
>
> Lucene50Codec is extremely central to almost everything the unreleased
> Lucene 5.x does if it is using default settings, so a very large number
> of tests would be affected by a change in the compressed stored field
> format.  Just for giggles, I grabbed a screenshot of the classes that
> directly reference Lucene50Codec:
>
> https://www.dropbox.com/s/eqq30uvckc8b19d/Lucene50Codec.png?dl=0

thank you very much for your insights. I did not realize that this
particular case has
such extensive dependencies
>
> The build system includes the ability to use Clover if it is present ...
> I don't know much about it, except that it's related to determining test
> coverage for parts of the software.  Perhaps a clover report could give
> you the info you need?
>
> https://www.atlassian.com/software/clover/overview

I have heard about Clover, but never used it. I will have a closer look
at it now
>
> Aside from that ... running the entire test suite multiple times
> whenever you change anything is the best option.  The test system
> randomizes a large number of options on each test run, so running the
> tests multiple times will test many different combinations.  A very
> large number of bugs have been discovered and fixed because of that
> randomization.


I think you are right that in certain cases, like for example in the
case of CompressingStoredFieldsFormat
it makes a lot of sense to run the entire test suit (even multiple times
considering randomization), but I would argue that there are probably
quite a few other cases where incremental testing would be sufficient,
at least as a first shot.

I am saying this, because I think for larger projects running entire
test suits do not scale well and that it is rather unefficient and
inconvenient for developers to run the entire test suite every time,
before they commit/push a tiny change.

Since I am not familiar enough with the code of Lucene/Solr I cannot say
whether all code is covered by tests,
but I would assume it is not. Also it would be interesting to know
whether developers do manual testing before they commit/push or whether
they just rely on running the entire test suit. I guess they are doing
manual tests as well, but probably only test things which they think are
related to the changes which they made.

Hence I think it would help developers to have a system which supports
incremental testing. I really do not mean to say that it should replace
the entire suite testing approach, but it would be additionally.

I will think about it some more :-)

Thanks

Michael

>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Michael Wechner <mi...@wyona.com>.
that is a great suggestion. Maybe the following failure of today

http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-MacOSX/1927/

and in particular

http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-MacOSX/1927/testReport/junit/org.apache.solr.cloud/OverseerStatusTest/testDistribSearch/

illustrates better why I think that we could be more efficient detecting issues and how this relates to your suggestion.

IIUC the build took 3 hours and 11 minutes on Mac OS X virtual box and then the failure happened. If we would have the association between the file changes and the tests, then we quite probably could detect such failures much quicker and hence get it fixed much quicker.


If there is no failure for such associated tests, then we could still run the whole test suite, to make sure
that we have not missed any dependencies for whatever reason.

I will try to get some prototype running, please apologize if I might ask stupid questions on my way.

Thanks

Michael



Am 03.12.14 um 15:08 schrieb Alexandre Rafalovitch:
> On 2 December 2014 at 18:10, Shawn Heisey <ap...@elyograg.org> wrote:
>> The test dependency tree for this particular class is *EXTENSIVE* and
>> not very easy to track down.
> How about a black box approach? Run each test individually once with a
> debugging interface enabled and the code that records every class load
> and every file system access. That's very similar to what JRebel does.
>
> Each test run is a Solr document. The loaded classes/files are
> multivalued fields. Dump into standalone Solr.
>
> Then, when the file changes, search by it's name/location and get back
> the list of tests that ever touched it.
>
> There are some complications due to randomised tests and also due to
> changed code possibly invoking new code path, but the core idea could
> probably be prototyped in a day or so to see if it has legts.
>
> Regards,
>    Alex.
>
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Alexandre Rafalovitch <ar...@gmail.com>.
On 2 December 2014 at 18:10, Shawn Heisey <ap...@elyograg.org> wrote:
> The test dependency tree for this particular class is *EXTENSIVE* and
> not very easy to track down.

How about a black box approach? Run each test individually once with a
debugging interface enabled and the code that records every class load
and every file system access. That's very similar to what JRebel does.

Each test run is a Solr document. The loaded classes/files are
multivalued fields. Dump into standalone Solr.

Then, when the file changes, search by it's name/location and get back
the list of tests that ever touched it.

There are some complications due to randomised tests and also due to
changed code possibly invoking new code path, but the core idea could
probably be prototyped in a day or so to see if it has legts.

Regards,
   Alex.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Running tests incrementally based on changed files

Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/2/2014 3:31 PM, Michael Wechner wrote:
> I am trying to setup a testing environment, which is running tests
> incrementally based on changed files, instead of running whole builds.
>
> I would like to try this for Lucene/Solr, hence I am trying to
> understand better the relationship between individual code changes and
> associated tests. For example I have noticed to the following change:
>
> git diff --name-only 641ed06..cb4fc89
> lucene/core/src/java/org/apache/lucene/codecs/compressing/CompressingStoredFieldsFormat.java
>
> and if I understand correctly, then the following tests would have to be executed:
>
> lucene/core/src/test/org/apache/lucene/codecs/compressing/TestCompressingStoredFieldsFormat.java
> lucene/test-framework/src/java/org/apache/lucene/codecs/compressing/CompressingCodec.java
>
> is that correct?

The test dependency tree for this particular class is *EXTENSIVE* and
not very easy to track down.

CompressingStoredFieldsFormat is the parent class for
Lucene50StoredFieldsFormat.  It is directly referenced by three classes
in addition to the direct descendant.  Here's a screenshot showing all
direct references in branch_5x:

https://www.dropbox.com/s/3pb4kljvdbd6xug/CompressingStoredFieldsFormat-eclipse.png?dl=0

Lucene50StoredFieldsFormat is used by a few classes, one of which is
Lucene50Codec:

https://www.dropbox.com/s/0ibnw87m6rveinv/Lucene50StoredFieldsFormat.png?dl=0

Lucene50Codec is extremely central to almost everything the unreleased
Lucene 5.x does if it is using default settings, so a very large number
of tests would be affected by a change in the compressed stored field
format.  Just for giggles, I grabbed a screenshot of the classes that
directly reference Lucene50Codec:

https://www.dropbox.com/s/eqq30uvckc8b19d/Lucene50Codec.png?dl=0

The build system includes the ability to use Clover if it is present ...
I don't know much about it, except that it's related to determining test
coverage for parts of the software.  Perhaps a clover report could give
you the info you need?

https://www.atlassian.com/software/clover/overview

Aside from that ... running the entire test suite multiple times
whenever you change anything is the best option.  The test system
randomizes a large number of options on each test run, so running the
tests multiple times will test many different combinations.  A very
large number of bugs have been discovered and fixed because of that
randomization.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org