You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (Created) (JIRA)" <ji...@apache.org> on 2012/03/24 13:12:26 UTC
[jira] [Created] (LUCENE-3910) remove special hudson nightly
linedocs
remove special hudson nightly linedocs
--------------------------------------
Key: LUCENE-3910
URL: https://issues.apache.org/jira/browse/LUCENE-3910
Project: Lucene - Java
Issue Type: Bug
Reporter: Robert Muir
Hudson has a special huge linedocs file that it sets via a -D parameter,
but this means that anything using LineDocs won't reproduce via our home
computers if it fails on hudson.
I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268297#comment-13268297 ]
Michael McCandless commented on LUCENE-3910:
--------------------------------------------
bq. If we are going to keep this large file, local reproducibility needs to be made easier.
+1
I think, first, we should add -Dtests.linedocsfile=XXX to the "reproduce line", when that was passed to 'ant test'.
Second, I think we should put the nightly line file somewhere "accessible". It's currently at http://people.apache.org/~mikemccand/enwiki.random.lines.txt ...
Maybe we can have an ant target, or ivy, to pull down a copy to your local area? I also like Dawid's idea to have a separate "big test data" repository somewhere...
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.1
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237925#comment-13237925 ]
Robert Muir commented on LUCENE-3910:
-------------------------------------
This one is controversial (at least Mike and I don't agree):
its a reproducibility versus coverage thing.
I'm gonna unset 3.6 because the problem already exists in other
3.x releases, and it only affects nightly builds: for end users
there is no concern.
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.0
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3910:
---------------------------------------
Attachment: LUCENE-3910.patch
Trivial patch that adds -Dtests.linedocsfile=XXX to the "reproduce with" line.
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.1
>
> Attachments: LUCENE-3910.patch
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268659#comment-13268659 ]
Michael McCandless commented on LUCENE-3910:
--------------------------------------------
Maybe as a baby step (before we figure out how to hold/release test data in a separate repository)... we can add ant/ivy sugar to pull down the nightly line docs file from p.a.o?
We can then eg improve the patch I put up to give instructions to run that...
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.1
>
> Attachments: LUCENE-3910.patch
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268305#comment-13268305 ]
Robert Muir commented on LUCENE-3910:
-------------------------------------
{quote}
I also like Dawid's idea to have a separate "big test data" repository somewhere...
{quote}
+1 for some better separation here. Source release and svn checkouts are bloated because of
all this test data.
E.g. linefile data is over 5MB, snowball test data is over 3MB, ...
Maybe we should separate this out in svn? its "artifact" is a .jar file
with all these huge files in the appropriate places or something?
So basically we could sucker this thing down with ivy and put it in the classpath like
any other dependency.
The problems that make it hard though, are versioning and "releasing" this thing:
* if its outside of dev/ SVN that makes versioning the testdata wrt different releases/branches hard.
this could easily get annoying and complicated.
* where would we put the resulting "jar" to download via ivy? I dont think we should be downloading
this from SVN, since our source releases would actually have this as a test dependency.
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.1
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3910:
---------------------------------------
Attachment: LUCENE-3910.patch
Improved patch adding the baby step: a new ant target "get-jenkins-line-docs", and an additional NOTE on a test failure that you can use this target to download the large line docs file.
I think it's ready...
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.1
>
> Attachments: LUCENE-3910.patch, LUCENE-3910.patch
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-3910.
----------------------------------------
Resolution: Fixed
Fix Version/s: 4.0
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.0, 4.1
>
> Attachments: LUCENE-3910.patch, LUCENE-3910.patch
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3910:
--------------------------------
Fix Version/s: 4.0
3.6
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3910:
--------------------------------
Fix Version/s: (was: 3.6)
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.0
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268252#comment-13268252 ]
Robert Muir commented on LUCENE-3910:
-------------------------------------
If we are going to keep this large file, local reproducibility needs to be made easier.
Currently, if something fails in the nightly build, nobody fixes it because of this problem:
I know i wont go wasting my time with tests that dont reproduce!
If we arent going to do this we should disable the linedocs (I will do this in 72 hours,
if the situation isnt improved)
There is absolutely no point of finding test fails that no one will debug because its too hard.
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.1
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3910) remove special hudson nightly
linedocs
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237944#comment-13237944 ]
Dawid Weiss commented on LUCENE-3910:
-------------------------------------
I agree with you both. No, it's not a paradox. On one hand -- I agree that having larger test files is good and on the other I agree with Robert that not being able to reproduce locally because of different (or inconsistent) data is a pain.
At Carrot Search we have put all the "big data" into a separate git repository and this is simply mirrored across build servers and our local machines. Granted, the first clone takes a while, but then pulls of additional data are much faster and (which is a big plus) git repo has an md5 of the revision so this can be emitted as a log upon failure (we don't do it because we're pretty much sure the checkouts are consistent, but it _could_ be done to ensure testing against exact same test files).
Just thoughts to consider.
> remove special hudson nightly linedocs
> --------------------------------------
>
> Key: LUCENE-3910
> URL: https://issues.apache.org/jira/browse/LUCENE-3910
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Robert Muir
> Fix For: 4.0
>
>
> Hudson has a special huge linedocs file that it sets via a -D parameter,
> but this means that anything using LineDocs won't reproduce via our home
> computers if it fails on hudson.
> I think we should disable this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org