You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Chris Hostetter <ho...@fucit.org> on 2014/12/01 17:46:08 UTC
Re: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b09)
- Build # 4460 - Still Failing!
: windowsfs uses map<inode,integer> to determine if it can delete a
: file, but this is troublesome on windows. One problem is that we cant
: 100% compare files for equality without doing i/o. And doing this
: itself can piss "the real windows" off. So I just disable windowsfs on
: actual windows for now, i don't really have a way to test it and i
: dont think it buys us anything right now to add lots of special code.
I've got a massive backlog of lucene/solr related email, and haven't
really been able to follow all the commits, so forgive me if this is a
stupid question but...
as i understand it, the test framework has a bunch of new stuff now for
mocking the filesystem (cool!) and this is the context for the "windowsfs"
refrence here (correct?) ... so my question is this:
when saying "disable windowsfs on actual windows" does that mean:
1: (mock) windowsfs is removed from the pool of random mockfs's that are
candidates for testing when we detect the OS is "windows"
...or does that mean that...
2: if the radomizer selects (mock) windowsfs *and* we detect the OS is
windows, then we ignore the "(mock) windowsfs" and just use the real
filesystem?
The reason i ask: both approaches seem ot have serious downsides, and i'm
trying to understand which devil we're choosing?
the downside for #2 is that a test on linux which cliams it's using
"(mock) windowsfs" might pass/fail diff then the same test with the same
seed on an actual windows machine where the fs is really diff --- but that
seems no worse then what we've had in the past before we made any attempt
to mock out hte filesystem.
the downside of #1 however seems to have a lot more serious impact on
people who are actualy running windows, and really want to test "how does
lucene/solr perform on windows" ... becuase they will *never* see their
tests run on a windows filesystem (either mock or real)
...or am i missunderstanding something fundemental?
:
:
: On Sun, Nov 30, 2014 at 5:31 AM, Michael McCandless
: <lu...@mikemccandless.com> wrote:
: > I think Rob fixed this with rev 1642486
: >
: > Mike McCandless
: >
: > http://blog.mikemccandless.com
: >
: >
: > On Sat, Nov 29, 2014 at 3:31 PM, Policeman Jenkins Server
: > <je...@thetaphi.de> wrote:
: >> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4460/
: >> Java: 64bit/jdk1.8.0_40-ea-b09 -XX:+UseCompressedOops -XX:+UseG1GC (asserts: true)
: >>
: >> 1 tests failed.
: >> FAILED: org.apache.lucene.util.junitcompat.TestLeaveFilesIfTestFails.testWindowsUnremovableFile
: >>
: >> Error Message:
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\core\test\J0\temp\lucene.util.junitcompat.TestLeaveFilesIfTestFails$Nested2-C5F2CA80130DA348-001\leftover-001\child.locked
: >>
: >> Stack Trace:
: >> java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\core\test\J0\temp\lucene.util.junitcompat.TestLeaveFilesIfTestFails$Nested2-C5F2CA80130DA348-001\leftover-001\child.locked
: >> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
: >> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
: >> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
: >> at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:53)
: >> at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:38)
: >> at org.apache.lucene.mockfile.WindowsFS.getKey(WindowsFS.java:57)
: >> at org.apache.lucene.mockfile.WindowsFS.onClose(WindowsFS.java:77)
: >> at org.apache.lucene.mockfile.HandleTrackingFS$5.close(HandleTrackingFS.java:210)
: >> at org.apache.lucene.util.junitcompat.TestLeaveFilesIfTestFails.testWindowsUnremovableFile(TestLeaveFilesIfTestFails.java:79)
: >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
: >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
: >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
: >> at java.lang.reflect.Method.invoke(Method.java:497)
: >> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
: >> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
: >> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
: >> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
: >> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
: >> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
: >> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
: >> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: >> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
: >> at org.junit.rules.RunRules.evaluate(RunRules.java:18)
: >> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
: >> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
: >> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
: >> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
: >> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
: >> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
: >> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
: >> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
: >> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: >> at org.junit.rules.RunRules.evaluate(RunRules.java:18)
: >> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
: >> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:199)
: >> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:310)
: >> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
: >>
: >>
: >>
: >>
: >> Build Log:
: >> [...truncated 436 lines...]
: >> [junit4] Suite: org.apache.lucene.util.junitcompat.TestLeaveFilesIfTestFails
: >> [junit4] 1> sysout from nested test:
: >> [junit4] 1>
: >> [junit4] 1> syserr from nested test: NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=sr_CS, timezone=Etc/UCT
: >> [junit4] 1> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=2,threads=1,free=81523472,total=214958080
: >> [junit4] 1> NOTE: All tests run in this JVM: [TestAutomaton, TestQueryBuilder, Test2BNumericDocValues, TestDirectory, TestDocumentsWriterDeleteQueue, TestBooleanOr, TestPositiveScoresOnlyCollector, TestMmapDirectory, TestBagOfPostings, TestOperations, TestFieldReuse, TestBooleanScorer, TestNoMergeScheduler, TestFieldMaskingSpanQuery, TestHighCompressionMode, TestFieldsReader, TestLucene50StoredFieldsFormat, TestFilterIterator, TestStressDeletes, TestIndexWriterOnDiskFull, TestForceMergeForever, MultiCollectorTest, TestNumericDocValuesUpdates, TestConsistentFieldNumbers, TestSimpleExplanationsOfNonMatches, TestPayloadExplanations, TestSegmentTermDocs, TestIsCurrent, TestNIOFSDirectory, TestMockAnalyzer, TestNumericRangeQuery32, TestMultiDocValues, TestLevenshteinAutomata, TestByteArrayDataInput, TestSortRandom, TestByteSlices, TestMultiMMap, Test2BTerms, TestIndexInput, TestDocValuesIndexing, TestDemoParallelLeafReader, TestAddIndexes, TestBoolean2, TestDemo, TestIndexWriter
Exceptions2, TestManyFields, TestDocumentsWriterStallControl, TestRegExp, TestFieldValueFilter, TestLogMergePolicy, TestSearcherManager, Nested1, Nested2]
: >> [junit4] 1> NOTE: reproduce with: ant test -Dtestcase=Nested2 -Dtests.seed=C5F2CA80130DA348 -Dtests.slow=true -Dtests.locale=sr_CS -Dtests.timezone=Etc/UCT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
: >> [junit4] 1>
: >> [junit4] ERROR 0.04s | TestLeaveFilesIfTestFails.testWindowsUnremovableFile <<<
: >> [junit4] > Throwable #1: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\core\test\J0\temp\lucene.util.junitcompat.TestLeaveFilesIfTestFails$Nested2-C5F2CA80130DA348-001\leftover-001\child.locked
: >> [junit4] > at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
: >> [junit4] > at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
: >> [junit4] > at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
: >> [junit4] > at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:53)
: >> [junit4] > at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:38)
: >> [junit4] > at org.apache.lucene.mockfile.WindowsFS.getKey(WindowsFS.java:57)
: >> [junit4] > at org.apache.lucene.mockfile.WindowsFS.onClose(WindowsFS.java:77)
: >> [junit4] > at org.apache.lucene.mockfile.HandleTrackingFS$5.close(HandleTrackingFS.java:210)
: >> [junit4] > at org.apache.lucene.util.junitcompat.TestLeaveFilesIfTestFails.testWindowsUnremovableFile(TestLeaveFilesIfTestFails.java:79)
: >> [junit4] Completed in 0.10s, 2 tests, 1 error <<< FAILURES!
: >>
: >> [...truncated 1156 lines...]
: >> BUILD FAILED
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:515: The following error occurred while executing this line:
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:463: The following error occurred while executing this line:
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:61: The following error occurred while executing this line:
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:49: The following error occurred while executing this line:
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1349: The following error occurred while executing this line:
: >> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:956: There were test failures: 405 suites, 3053 tests, 1 error, 63 ignored (53 assumptions)
: >>
: >> Total time: 10 minutes 43 seconds
: >> Build step 'Invoke Ant' marked build as failure
: >> [description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b09 -XX:+UseCompressedOops -XX:+UseG1GC (asserts: true)
: >> Archiving artifacts
: >> Recording test results
: >> Email was triggered for: Failure - Any
: >> Sending email for trigger: Failure - Any
: >>
: >>
: >>
: >>
: >> ---------------------------------------------------------------------
: >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
: >> For additional commands, e-mail: dev-help@lucene.apache.org
: >
: > ---------------------------------------------------------------------
: > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
: > For additional commands, e-mail: dev-help@lucene.apache.org
: >
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
: For additional commands, e-mail: dev-help@lucene.apache.org
:
:
-Hoss
http://www.lucidworks.com/
Re: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b09) -
Build # 4460 - Still Failing!
Posted by Robert Muir <rc...@gmail.com>.
On Mon, Dec 1, 2014 at 12:24 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> right, i guess a better way to restate my question -- now that i
> understand better what is being "mocked" -- is to ask:
>
> Shouldn't there an easy way (ie: sys prop?) for end users, who may
> primarily be interested in testing lucene/solr on *their* OS with *their*
> filesystem, to tell the test framework to skip all mocking of the
> filesystem?
>
> so if someone wants to try out a patch, or do regression testing before
> upgrading, and wants to beast all the lucene/solr tests on their
> fs/os/hardware stack can force all X runs to use their *real* filesystem
> w/o any mocking?
>
I dont think so.
Such a thing is unrelated to this change. We have had "act like
windows" and other filesystem-mocking code in our tests forever
(MockDirectoryWrapper), and we add lots of asserting-wrappers and
other mocks around the codec API and other key places during tests.
This change just allows us to mock filesystem behavior underneath the
Directory api, and its usable for testing things against filesystem
operations that don't go thru the directory api too.
The purpose of our tests is to ensure the code is correct and nothing more.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b09)
- Build # 4460 - Still Failing!
Posted by Chris Hostetter <ho...@fucit.org>.
: Its easiest to explain i think by pasting the code. for 10% of runs we
: don't wrap the filesystem at all. This is to ensure our "mock
Ah - ok. the 10% use "real" filesystem is the biggest part i was missing.
So 90% of the time we wrap the filesystem in special checks; and 10% of
the time when you do special checks we emulate windows *unless* we're
already on windows.
: order to prevent that, we would have to do a whole lot more, including
: "mocking linux" on windows. But thats just a non-goal here. These
: mockfilesystems do not make it any better or any worse.
yeah -- i missunderstood the scope of what was being mocked.
: If you are running on windows, then your actual filesystem is already
: acting like windows. We don't need to add any code to accomplish
: "windows behavior" :)
right, i guess a better way to restate my question -- now that i
understand better what is being "mocked" -- is to ask:
Shouldn't there an easy way (ie: sys prop?) for end users, who may
primarily be interested in testing lucene/solr on *their* OS with *their*
filesystem, to tell the test framework to skip all mocking of the
filesystem?
so if someone wants to try out a patch, or do regression testing before
upgrading, and wants to beast all the lucene/solr tests on their
fs/os/hardware stack can force all X runs to use their *real* filesystem
w/o any mocking?
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
: For additional commands, e-mail: dev-help@lucene.apache.org
:
:
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b09) -
Build # 4460 - Still Failing!
Posted by Robert Muir <rc...@gmail.com>.
On Mon, Dec 1, 2014 at 11:46 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : windowsfs uses map<inode,integer> to determine if it can delete a
> : file, but this is troublesome on windows. One problem is that we cant
> : 100% compare files for equality without doing i/o. And doing this
> : itself can piss "the real windows" off. So I just disable windowsfs on
> : actual windows for now, i don't really have a way to test it and i
> : dont think it buys us anything right now to add lots of special code.
>
> I've got a massive backlog of lucene/solr related email, and haven't
> really been able to follow all the commits, so forgive me if this is a
> stupid question but...
>
> as i understand it, the test framework has a bunch of new stuff now for
> mocking the filesystem (cool!) and this is the context for the "windowsfs"
> refrence here (correct?) ... so my question is this:
>
> when saying "disable windowsfs on actual windows" does that mean:
>
> 1: (mock) windowsfs is removed from the pool of random mockfs's that are
> candidates for testing when we detect the OS is "windows"
> ...or does that mean that...
Its easiest to explain i think by pasting the code. for 10% of runs we
don't wrap the filesystem at all. This is to ensure our "mock
filesystems" dont hide real bugs.
Otherwise, we add some minimal checks: we disable actual fsync calls
(its unnecessary), fail on leaked file handles, limit the number of
open files to 2048 (currently).
Then we occasionally act like windows, if we are on a non-windows machine.
// sometimes just use a bare filesystem
if (random.nextInt(10) > 0) {
fs = new DisableFsyncFS(fs).getFileSystem(null);
fs = new LeakFS(fs).getFileSystem(null);
fs = new HandleLimitFS(fs, MAX_OPEN_FILES).getFileSystem(null);
// windows is currently slow
if (random.nextInt(10) == 0) {
// don't try to emulate windows on windows: they don't get along
if (!Constants.WINDOWS) {
fs = new WindowsFS(fs).getFileSystem(null);
}
}
}
>
> The reason i ask: both approaches seem ot have serious downsides, and i'm
> trying to understand which devil we're choosing?
>
> the downside for #2 is that a test on linux which cliams it's using
> "(mock) windowsfs" might pass/fail diff then the same test with the same
> seed on an actual windows machine where the fs is really diff --- but that
> seems no worse then what we've had in the past before we made any attempt
> to mock out hte filesystem.
This is not something I am currently trying to address with tests. In
general, tests might fail on windows and pass on linux and so on. In
order to prevent that, we would have to do a whole lot more, including
"mocking linux" on windows. But thats just a non-goal here. These
mockfilesystems do not make it any better or any worse.
>
> the downside of #1 however seems to have a lot more serious impact on
> people who are actualy running windows, and really want to test "how does
> lucene/solr perform on windows" ... becuase they will *never* see their
> tests run on a windows filesystem (either mock or real)
>
If you are running on windows, then your actual filesystem is already
acting like windows. We don't need to add any code to accomplish
"windows behavior" :)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org