You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Ishan Chattopadhyaya <ic...@gmail.com> on 2015/05/23 07:10:41 UTC
RE: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Not sure if it is related or helpful, but while debugging tests for SOLR-7468 yesterday, I encountered this
java.lang.NoSuchFieldError:
totalTermCount
few times, I had to forcefully clean at root of the project and it worked. I remember Anshum had to do that clean thing more than once to make it work and he remarked "don't ask why".
Sent from my Windows Phone
-----Original Message-----
From: "Erick Erickson" <er...@gmail.com>
Sent: 5/23/2015 6:15 AM
To: "dev@lucene.apache.org" <de...@lucene.apache.org>
Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build #12782 - Failure!
OK, this is somewhat weird. I still have the original tree that I
checked in from which was up to date before I committed the code and
the tests run from there fine. But a current trunk fails every time.
Now, the machine it works on is my Mac Pro, and the failures are on my
MacBook so there may be something going on there.
I've got to leave for a while, I'll copy the tree that works on the
Pro, update the copy and see if this test fails when I get back. If
they fail, I can diff the trees to see what changed and see if I can
make any sense out of this.
I can always @Ignore this test to cut down on the noise, probably do
that tonight if I don't have any revelations.
I see this stack trace which makes no sense to me whatsoever (see the
lines with lots of ***** in front). I looked at where the code
originates (BufferedUpdatesStream[277]) and it looks like this:
if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
And it's telling me there's no such field? Whaaaaatttt????
Which is freaking me out since I don't see how this would trigger the
exception. Is this a red herring? And, of course, this doesn't fail in
IntelliJ but it does fail every time from the shell. Siiiigggghhh.
Of course if this were something fundamental to Lucene, it seems like
this would be failing all over the place so I assume it's something to
do with CDCR... But what do I know?
1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
status=0 QTime=8
************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
c:source_collection s:shard1 r:core_node3
x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
totalTermCount
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2> at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[junit4] 2> at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
************* [junit4] 2> at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
[junit4] 2> at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
[junit4] 2> at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
[junit4] 2> at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
[junit4] 2> at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
[junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
[junit4] 2> ... 23 more
On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com> wrote:
> Looking......
>
>
> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>
>> 4 tests failed.
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> Some resources were not closed, shutdown, or released.
>>
>> Stack Trace:
>> java.lang.AssertionError: Some resources were not closed, shutdown, or released.
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at org.junit.Assert.assertTrue(Assert.java:43)
>> at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.ut
[The entire original message is not included.]
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Mark Miller <ma...@gmail.com>.
Hit this a while ago, simply cleaned up and it went away...
On Sun, Jun 14, 2015 at 9:34 PM Noble Paul <no...@gmail.com> wrote:
> This is a problem still.
>
> I made some changes to my trunk and I get this error always
>
> java.lang.NoSuchFieldError: totalTermCount
>
>
> This gives a false alarm and we waste time. We need to get to the
> bottom of this
>
> On Tue, May 26, 2015 at 8:58 PM, Erick Erickson <er...@gmail.com>
> wrote:
> > Yep. I found this while trying to deal with CDCR, which is only in trunk.
> >
> >
> >
> > On Mon, May 25, 2015 at 11:51 PM, Noble Paul <no...@gmail.com>
> wrote:
> >> Do you mean in trunk ?
> >>
> >> On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <
> erickerickson@gmail.com> wrote:
> >>> The problem went away for me. I took sledgehammer approach and deleted
> >>> the entire tree locally and checked out the tree fresh.
> >>>
> >>> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com>
> wrote:
> >>>> I'v edone a clean checkout I still see these errors
> >>>>
> >>>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> >>>>
> >>>> <title>Error 500 Server Error</title>
> >>>>
> >>>> </head>
> >>>>
> >>>> <body><h2>HTTP ERROR 500</h2>
> >>>>
> >>>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
> >>>>
> >>>> <pre> Server Error</pre></p><h3>Caused
> >>>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
> >>>>
> >>>> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
> >>>>
> >>>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
> >>>>
> >>>> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
> >>>>
> >>>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
> >>>> <er...@gmail.com> wrote:
> >>>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
> >>>>> everything just fine, my laptop fails all the time.
> >>>>>
> >>>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
> >>>>> <ic...@gmail.com> wrote:
> >>>>>> Not sure if it is related or helpful, but while debugging tests for
> >>>>>> SOLR-7468 yesterday, I encountered this
> >>>>>> java.lang.NoSuchFieldError:
> >>>>>> totalTermCount
> >>>>>>
> >>>>>> few times, I had to forcefully clean at root of the project and it
> worked. I
> >>>>>> remember Anshum had to do that clean thing more than once to make
> it work
> >>>>>> and he remarked "don't ask why".
> >>>>>>
> >>>>>> Sent from my Windows Phone
> >>>>>> ________________________________
> >>>>>> From: Erick Erickson
> >>>>>> Sent: 5/23/2015 6:15 AM
> >>>>>> To: dev@lucene.apache.org
> >>>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45)
> - Build
> >>>>>> #12782 - Failure!
> >>>>>>
> >>>>>> OK, this is somewhat weird. I still have the original tree that I
> >>>>>> checked in from which was up to date before I committed the code and
> >>>>>> the tests run from there fine. But a current trunk fails every time.
> >>>>>> Now, the machine it works on is my Mac Pro, and the failures are on
> my
> >>>>>> MacBook so there may be something going on there.
> >>>>>>
> >>>>>> I've got to leave for a while, I'll copy the tree that works on the
> >>>>>> Pro, update the copy and see if this test fails when I get back. If
> >>>>>> they fail, I can diff the trees to see what changed and see if I can
> >>>>>> make any sense out of this.
> >>>>>>
> >>>>>> I can always @Ignore this test to cut down on the noise, probably do
> >>>>>> that tonight if I don't have any revelations.
> >>>>>>
> >>>>>> I see this stack trace which makes no sense to me whatsoever (see
> the
> >>>>>> lines with lots of ***** in front). I looked at where the code
> >>>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
> >>>>>>
> >>>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount !=
> 0) {
> >>>>>>
> >>>>>> And it's telling me there's no such field? Whaaaaatttt????
> >>>>>>
> >>>>>> Which is freaking me out since I don't see how this would trigger
> the
> >>>>>> exception. Is this a red herring? And, of course, this doesn't fail
> in
> >>>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
> >>>>>>
> >>>>>> Of course if this were something fundamental to Lucene, it seems
> like
> >>>>>> this would be failing all over the place so I assume it's something
> to
> >>>>>> do with CDCR... But what do I know?
> >>>>>>
> >>>>>>
> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> >>>>>> status=0 QTime=8
> >>>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
> >>>>>> c:source_collection s:shard1 r:core_node3
> >>>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log
> ERROR
> >>>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
> >>>>>> totalTermCount
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
> >>>>>> [junit4] 2> at
> >>>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >>>>>> [junit4] 2> at
> org.eclipse.jetty.server.Server.handle(Server.java:497)
> >>>>>> [junit4] 2> at
> >>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> >>>>>> [junit4] 2> at
> >>>>>> org.eclipse.jetty.io
> .AbstractConnection$2.run(AbstractConnection.java:540)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> >>>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
> >>>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError:
> totalTermCount
> >>>>>> ************* [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
> >>>>>> [junit4] 2> ... 23 more
> >>>>>>
> >>>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <
> erickerickson@gmail.com>
> >>>>>> wrote:
> >>>>>>> Looking......
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
> >>>>>>> <je...@thetaphi.de> wrote:
> >>>>>>>> Build:
> http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
> >>>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops
> -XX:+UseConcMarkSweepGC
> >>>>>>>>
> >>>>>>>> 4 tests failed.
> >>>>>>>> FAILED:
> >>>>>>>> junit.framework.TestSuite.org
> .apache.solr.cloud.CdcrReplicationDistributedZkTest
> >>>>>>>>
> >>>>>>>> Error Message:
> >>>>>>>> Some resources were not closed, shutdown, or released.
> >>>>>>>>
> >>>>>>>> Stack Trace:
> >>>>>>>> java.lang.AssertionError: Some resources were not closed,
> shutdown, or
> >>>>>>>> released.
> >>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
> >>>>>>>> at org.junit.Assert.fail(Assert.java:93)
> >>>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
> >>>>>>>> at
> >>>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
> >>>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown
> Source)
> >>>>>>>> at
> >>>>>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> FAILED:
> >>>>>>>> junit.framework.TestSuite.org
> .apache.solr.cloud.CdcrReplicationDistributedZkTest
> >>>>>>>>
> >>>>>>>> Error Message:
> >>>>>>>> 5 threads leaked from SUITE scope at
> >>>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
> >>>>>>>> Thread[id=4675,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-SendThread(127.0.0.1:54300),
> >>>>>>>> state=TIMED_WAITING,
> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> java.lang.Thread.sleep(Native Method) at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
> >>>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-EventThread,
> >>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> >>>>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>>>>> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> >>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
> >>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>
> >>>>>>>> Stack Trace:
> >>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads
> leaked from
> >>>>>>>> SUITE scope at
> org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
> >>>>>>>> 1) Thread[id=4675,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-SendThread(127.0.0.1:54300),
> >>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at java.lang.Thread.sleep(Native Method)
> >>>>>>>> at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
> >>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>> 3) Thread[id=4676,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-EventThread,
> >>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>>>>> at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> >>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> FAILED:
> >>>>>>>> junit.framework.TestSuite.org
> .apache.solr.cloud.CdcrReplicationDistributedZkTest
> >>>>>>>>
> >>>>>>>> Error Message:
> >>>>>>>> There are still zombie threads that couldn't be terminated: 1)
> >>>>>>>> Thread[id=4675,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-SendThread(127.0.0.1:54300),
> >>>>>>>> state=TIMED_WAITING,
> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> java.lang.Thread.sleep(Native Method) at
> >>>>>>>>
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> >>>>>>>> at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> >>>>>>>> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> >>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
> >>>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
> >>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at java.ut
> >>>>>>
> >>>>>> [The entire original message is not included.]
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> -----------------------------------------------------
> >>>> Noble Paul
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>
> >>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
- Mark
about.me/markrmiller
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Noble Paul <no...@gmail.com>.
This is a problem still.
I made some changes to my trunk and I get this error always
java.lang.NoSuchFieldError: totalTermCount
This gives a false alarm and we waste time. We need to get to the
bottom of this
On Tue, May 26, 2015 at 8:58 PM, Erick Erickson <er...@gmail.com> wrote:
> Yep. I found this while trying to deal with CDCR, which is only in trunk.
>
>
>
> On Mon, May 25, 2015 at 11:51 PM, Noble Paul <no...@gmail.com> wrote:
>> Do you mean in trunk ?
>>
>> On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <er...@gmail.com> wrote:
>>> The problem went away for me. I took sledgehammer approach and deleted
>>> the entire tree locally and checked out the tree fresh.
>>>
>>> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
>>>> I'v edone a clean checkout I still see these errors
>>>>
>>>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>>>>
>>>> <title>Error 500 Server Error</title>
>>>>
>>>> </head>
>>>>
>>>> <body><h2>HTTP ERROR 500</h2>
>>>>
>>>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>>>>
>>>> <pre> Server Error</pre></p><h3>Caused
>>>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>>>>
>>>> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>>
>>>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>>
>>>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
>>>> <er...@gmail.com> wrote:
>>>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>>>>> everything just fine, my laptop fails all the time.
>>>>>
>>>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>>>>> <ic...@gmail.com> wrote:
>>>>>> Not sure if it is related or helpful, but while debugging tests for
>>>>>> SOLR-7468 yesterday, I encountered this
>>>>>> java.lang.NoSuchFieldError:
>>>>>> totalTermCount
>>>>>>
>>>>>> few times, I had to forcefully clean at root of the project and it worked. I
>>>>>> remember Anshum had to do that clean thing more than once to make it work
>>>>>> and he remarked "don't ask why".
>>>>>>
>>>>>> Sent from my Windows Phone
>>>>>> ________________________________
>>>>>> From: Erick Erickson
>>>>>> Sent: 5/23/2015 6:15 AM
>>>>>> To: dev@lucene.apache.org
>>>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>>>>> #12782 - Failure!
>>>>>>
>>>>>> OK, this is somewhat weird. I still have the original tree that I
>>>>>> checked in from which was up to date before I committed the code and
>>>>>> the tests run from there fine. But a current trunk fails every time.
>>>>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>>>>> MacBook so there may be something going on there.
>>>>>>
>>>>>> I've got to leave for a while, I'll copy the tree that works on the
>>>>>> Pro, update the copy and see if this test fails when I get back. If
>>>>>> they fail, I can diff the trees to see what changed and see if I can
>>>>>> make any sense out of this.
>>>>>>
>>>>>> I can always @Ignore this test to cut down on the noise, probably do
>>>>>> that tonight if I don't have any revelations.
>>>>>>
>>>>>> I see this stack trace which makes no sense to me whatsoever (see the
>>>>>> lines with lots of ***** in front). I looked at where the code
>>>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>>>>
>>>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>>>>
>>>>>> And it's telling me there's no such field? Whaaaaatttt????
>>>>>>
>>>>>> Which is freaking me out since I don't see how this would trigger the
>>>>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>>>>
>>>>>> Of course if this were something fundamental to Lucene, it seems like
>>>>>> this would be failing all over the place so I assume it's something to
>>>>>> do with CDCR... But what do I know?
>>>>>>
>>>>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>>>>> status=0 QTime=8
>>>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>>>>> c:source_collection s:shard1 r:core_node3
>>>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>>>>> totalTermCount
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>>>>> ************* [junit4] 2> at
>>>>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>>>>> [junit4] 2> ... 23 more
>>>>>>
>>>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>>>>> wrote:
>>>>>>> Looking......
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>>>>> <je...@thetaphi.de> wrote:
>>>>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>>>>
>>>>>>>> 4 tests failed.
>>>>>>>> FAILED:
>>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>>
>>>>>>>> Error Message:
>>>>>>>> Some resources were not closed, shutdown, or released.
>>>>>>>>
>>>>>>>> Stack Trace:
>>>>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>>>>> released.
>>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>>>>> at
>>>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>
>>>>>>>>
>>>>>>>> FAILED:
>>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>>
>>>>>>>> Error Message:
>>>>>>>> 5 threads leaked from SUITE scope at
>>>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>>>>> Thread[id=4675,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>>> at
>>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>
>>>>>>>> Stack Trace:
>>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>>>>> 1) Thread[id=4675,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at java.lang.Thread.sleep(Native Method)
>>>>>>>> at
>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> 3) Thread[id=4676,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>>> at
>>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>>> at
>>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>>>
>>>>>>>>
>>>>>>>> FAILED:
>>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>>
>>>>>>>> Error Message:
>>>>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>>>>> Thread[id=4675,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>>>>> at
>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.ut
>>>>>>
>>>>>> [The entire original message is not included.]
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -----------------------------------------------------
>>>> Noble Paul
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
Yep. I found this while trying to deal with CDCR, which is only in trunk.
On Mon, May 25, 2015 at 11:51 PM, Noble Paul <no...@gmail.com> wrote:
> Do you mean in trunk ?
>
> On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <er...@gmail.com> wrote:
>> The problem went away for me. I took sledgehammer approach and deleted
>> the entire tree locally and checked out the tree fresh.
>>
>> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
>>> I'v edone a clean checkout I still see these errors
>>>
>>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>>>
>>> <title>Error 500 Server Error</title>
>>>
>>> </head>
>>>
>>> <body><h2>HTTP ERROR 500</h2>
>>>
>>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>>>
>>> <pre> Server Error</pre></p><h3>Caused
>>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>>>
>>> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>
>>> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>
>>> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>
>>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>
>>> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>
>>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>
>>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>
>>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
>>> <er...@gmail.com> wrote:
>>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>>>> everything just fine, my laptop fails all the time.
>>>>
>>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>>>> <ic...@gmail.com> wrote:
>>>>> Not sure if it is related or helpful, but while debugging tests for
>>>>> SOLR-7468 yesterday, I encountered this
>>>>> java.lang.NoSuchFieldError:
>>>>> totalTermCount
>>>>>
>>>>> few times, I had to forcefully clean at root of the project and it worked. I
>>>>> remember Anshum had to do that clean thing more than once to make it work
>>>>> and he remarked "don't ask why".
>>>>>
>>>>> Sent from my Windows Phone
>>>>> ________________________________
>>>>> From: Erick Erickson
>>>>> Sent: 5/23/2015 6:15 AM
>>>>> To: dev@lucene.apache.org
>>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>>>> #12782 - Failure!
>>>>>
>>>>> OK, this is somewhat weird. I still have the original tree that I
>>>>> checked in from which was up to date before I committed the code and
>>>>> the tests run from there fine. But a current trunk fails every time.
>>>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>>>> MacBook so there may be something going on there.
>>>>>
>>>>> I've got to leave for a while, I'll copy the tree that works on the
>>>>> Pro, update the copy and see if this test fails when I get back. If
>>>>> they fail, I can diff the trees to see what changed and see if I can
>>>>> make any sense out of this.
>>>>>
>>>>> I can always @Ignore this test to cut down on the noise, probably do
>>>>> that tonight if I don't have any revelations.
>>>>>
>>>>> I see this stack trace which makes no sense to me whatsoever (see the
>>>>> lines with lots of ***** in front). I looked at where the code
>>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>>>
>>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>>>
>>>>> And it's telling me there's no such field? Whaaaaatttt????
>>>>>
>>>>> Which is freaking me out since I don't see how this would trigger the
>>>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>>>
>>>>> Of course if this were something fundamental to Lucene, it seems like
>>>>> this would be failing all over the place so I assume it's something to
>>>>> do with CDCR... But what do I know?
>>>>>
>>>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>>>> status=0 QTime=8
>>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>>>> c:source_collection s:shard1 r:core_node3
>>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>>>> totalTermCount
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>>>> ************* [junit4] 2> at
>>>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>>>> [junit4] 2> ... 23 more
>>>>>
>>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>>>> wrote:
>>>>>> Looking......
>>>>>>
>>>>>>
>>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>>>> <je...@thetaphi.de> wrote:
>>>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>>>
>>>>>>> 4 tests failed.
>>>>>>> FAILED:
>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>
>>>>>>> Error Message:
>>>>>>> Some resources were not closed, shutdown, or released.
>>>>>>>
>>>>>>> Stack Trace:
>>>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>>>> released.
>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>>>> at
>>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>>>> at
>>>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>>
>>>>>>> FAILED:
>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>
>>>>>>> Error Message:
>>>>>>> 5 threads leaked from SUITE scope at
>>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>>>> Thread[id=4675,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>> at
>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> Stack Trace:
>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>>>> 1) Thread[id=4675,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at java.lang.Thread.sleep(Native Method)
>>>>>>> at
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 3) Thread[id=4676,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>> at
>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>> at
>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>>
>>>>>>>
>>>>>>> FAILED:
>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>
>>>>>>> Error Message:
>>>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>>>> Thread[id=4675,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>>>> at
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.ut
>>>>>
>>>>> [The entire original message is not included.]
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Noble Paul <no...@gmail.com>.
Do you mean in trunk ?
On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <er...@gmail.com> wrote:
> The problem went away for me. I took sledgehammer approach and deleted
> the entire tree locally and checked out the tree fresh.
>
> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
>> I'v edone a clean checkout I still see these errors
>>
>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>>
>> <title>Error 500 Server Error</title>
>>
>> </head>
>>
>> <body><h2>HTTP ERROR 500</h2>
>>
>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>>
>> <pre> Server Error</pre></p><h3>Caused
>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>>
>> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>
>> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>
>> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>
>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>
>> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>
>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>
>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>
>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
>> <er...@gmail.com> wrote:
>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>>> everything just fine, my laptop fails all the time.
>>>
>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>>> <ic...@gmail.com> wrote:
>>>> Not sure if it is related or helpful, but while debugging tests for
>>>> SOLR-7468 yesterday, I encountered this
>>>> java.lang.NoSuchFieldError:
>>>> totalTermCount
>>>>
>>>> few times, I had to forcefully clean at root of the project and it worked. I
>>>> remember Anshum had to do that clean thing more than once to make it work
>>>> and he remarked "don't ask why".
>>>>
>>>> Sent from my Windows Phone
>>>> ________________________________
>>>> From: Erick Erickson
>>>> Sent: 5/23/2015 6:15 AM
>>>> To: dev@lucene.apache.org
>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>>> #12782 - Failure!
>>>>
>>>> OK, this is somewhat weird. I still have the original tree that I
>>>> checked in from which was up to date before I committed the code and
>>>> the tests run from there fine. But a current trunk fails every time.
>>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>>> MacBook so there may be something going on there.
>>>>
>>>> I've got to leave for a while, I'll copy the tree that works on the
>>>> Pro, update the copy and see if this test fails when I get back. If
>>>> they fail, I can diff the trees to see what changed and see if I can
>>>> make any sense out of this.
>>>>
>>>> I can always @Ignore this test to cut down on the noise, probably do
>>>> that tonight if I don't have any revelations.
>>>>
>>>> I see this stack trace which makes no sense to me whatsoever (see the
>>>> lines with lots of ***** in front). I looked at where the code
>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>>
>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>>
>>>> And it's telling me there's no such field? Whaaaaatttt????
>>>>
>>>> Which is freaking me out since I don't see how this would trigger the
>>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>>
>>>> Of course if this were something fundamental to Lucene, it seems like
>>>> this would be failing all over the place so I assume it's something to
>>>> do with CDCR... But what do I know?
>>>>
>>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>>> status=0 QTime=8
>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>>> c:source_collection s:shard1 r:core_node3
>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>>> totalTermCount
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>> [junit4] 2> at
>>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>>> ************* [junit4] 2> at
>>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>>> [junit4] 2> at
>>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>>> [junit4] 2> at
>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>>> [junit4] 2> at
>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>>> [junit4] 2> at
>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>>> [junit4] 2> ... 23 more
>>>>
>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>>> wrote:
>>>>> Looking......
>>>>>
>>>>>
>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>>> <je...@thetaphi.de> wrote:
>>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>>
>>>>>> 4 tests failed.
>>>>>> FAILED:
>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>
>>>>>> Error Message:
>>>>>> Some resources were not closed, shutdown, or released.
>>>>>>
>>>>>> Stack Trace:
>>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>>> released.
>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>>> at
>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>>> at
>>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>> FAILED:
>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>
>>>>>> Error Message:
>>>>>> 5 threads leaked from SUITE scope at
>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>>> Thread[id=4675,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>> at
>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> Stack Trace:
>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>>> 1) Thread[id=4675,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at java.lang.Thread.sleep(Native Method)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 3) Thread[id=4676,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>>> at
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>> at
>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>
>>>>>>
>>>>>> FAILED:
>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>
>>>>>> Error Message:
>>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>>> Thread[id=4675,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.ut
>>>>
>>>> [The entire original message is not included.]
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
The problem went away for me. I took sledgehammer approach and deleted
the entire tree locally and checked out the tree fresh.
On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
> I'v edone a clean checkout I still see these errors
>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>
> <title>Error 500 Server Error</title>
>
> </head>
>
> <body><h2>HTTP ERROR 500</h2>
>
> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>
> <pre> Server Error</pre></p><h3>Caused
> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>
> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>
> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>
> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>
> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>
> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>
> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>
> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
> <er...@gmail.com> wrote:
>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>> everything just fine, my laptop fails all the time.
>>
>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>> <ic...@gmail.com> wrote:
>>> Not sure if it is related or helpful, but while debugging tests for
>>> SOLR-7468 yesterday, I encountered this
>>> java.lang.NoSuchFieldError:
>>> totalTermCount
>>>
>>> few times, I had to forcefully clean at root of the project and it worked. I
>>> remember Anshum had to do that clean thing more than once to make it work
>>> and he remarked "don't ask why".
>>>
>>> Sent from my Windows Phone
>>> ________________________________
>>> From: Erick Erickson
>>> Sent: 5/23/2015 6:15 AM
>>> To: dev@lucene.apache.org
>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>> #12782 - Failure!
>>>
>>> OK, this is somewhat weird. I still have the original tree that I
>>> checked in from which was up to date before I committed the code and
>>> the tests run from there fine. But a current trunk fails every time.
>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>> MacBook so there may be something going on there.
>>>
>>> I've got to leave for a while, I'll copy the tree that works on the
>>> Pro, update the copy and see if this test fails when I get back. If
>>> they fail, I can diff the trees to see what changed and see if I can
>>> make any sense out of this.
>>>
>>> I can always @Ignore this test to cut down on the noise, probably do
>>> that tonight if I don't have any revelations.
>>>
>>> I see this stack trace which makes no sense to me whatsoever (see the
>>> lines with lots of ***** in front). I looked at where the code
>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>
>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>
>>> And it's telling me there's no such field? Whaaaaatttt????
>>>
>>> Which is freaking me out since I don't see how this would trigger the
>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>
>>> Of course if this were something fundamental to Lucene, it seems like
>>> this would be failing all over the place so I assume it's something to
>>> do with CDCR... But what do I know?
>>>
>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>> status=0 QTime=8
>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>> c:source_collection s:shard1 r:core_node3
>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>> totalTermCount
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>> [junit4] 2> at
>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>> [junit4] 2> at
>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>> [junit4] 2> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>> [junit4] 2> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>> ************* [junit4] 2> at
>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>> [junit4] 2> at
>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>> [junit4] 2> at
>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>> [junit4] 2> at
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>> [junit4] 2> at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>> [junit4] 2> at
>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>> [junit4] 2> ... 23 more
>>>
>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>> wrote:
>>>> Looking......
>>>>
>>>>
>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>> <je...@thetaphi.de> wrote:
>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>
>>>>> 4 tests failed.
>>>>> FAILED:
>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>
>>>>> Error Message:
>>>>> Some resources were not closed, shutdown, or released.
>>>>>
>>>>> Stack Trace:
>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>> released.
>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>> at
>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>> at
>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>>
>>>>> FAILED:
>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>
>>>>> Error Message:
>>>>> 5 threads leaked from SUITE scope at
>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>> Thread[id=4675,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> java.lang.Thread.sleep(Native Method) at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> Stack Trace:
>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>> 1) Thread[id=4675,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> 3) Thread[id=4676,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>
>>>>>
>>>>> FAILED:
>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>
>>>>> Error Message:
>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>> Thread[id=4675,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> java.lang.Thread.sleep(Native Method) at
>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.ut
>>>
>>> [The entire original message is not included.]
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Noble Paul <no...@gmail.com>.
I'v edone a clean checkout I still see these errors
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
<pre> Server Error</pre></p><h3>Caused
by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
<er...@gmail.com> wrote:
> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
> everything just fine, my laptop fails all the time.
>
> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
> <ic...@gmail.com> wrote:
>> Not sure if it is related or helpful, but while debugging tests for
>> SOLR-7468 yesterday, I encountered this
>> java.lang.NoSuchFieldError:
>> totalTermCount
>>
>> few times, I had to forcefully clean at root of the project and it worked. I
>> remember Anshum had to do that clean thing more than once to make it work
>> and he remarked "don't ask why".
>>
>> Sent from my Windows Phone
>> ________________________________
>> From: Erick Erickson
>> Sent: 5/23/2015 6:15 AM
>> To: dev@lucene.apache.org
>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>> #12782 - Failure!
>>
>> OK, this is somewhat weird. I still have the original tree that I
>> checked in from which was up to date before I committed the code and
>> the tests run from there fine. But a current trunk fails every time.
>> Now, the machine it works on is my Mac Pro, and the failures are on my
>> MacBook so there may be something going on there.
>>
>> I've got to leave for a while, I'll copy the tree that works on the
>> Pro, update the copy and see if this test fails when I get back. If
>> they fail, I can diff the trees to see what changed and see if I can
>> make any sense out of this.
>>
>> I can always @Ignore this test to cut down on the noise, probably do
>> that tonight if I don't have any revelations.
>>
>> I see this stack trace which makes no sense to me whatsoever (see the
>> lines with lots of ***** in front). I looked at where the code
>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>
>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>
>> And it's telling me there's no such field? Whaaaaatttt????
>>
>> Which is freaking me out since I don't see how this would trigger the
>> exception. Is this a red herring? And, of course, this doesn't fail in
>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>
>> Of course if this were something fundamental to Lucene, it seems like
>> this would be failing all over the place so I assume it's something to
>> do with CDCR... But what do I know?
>>
>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>> status=0 QTime=8
>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>> c:source_collection s:shard1 r:core_node3
>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>> totalTermCount
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>> [junit4] 2> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>> [junit4] 2> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> [junit4] 2> at
>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> [junit4] 2> at
>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>> [junit4] 2> at
>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> [junit4] 2> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> [junit4] 2> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>> [junit4] 2> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> [junit4] 2> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> [junit4] 2> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> [junit4] 2> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> [junit4] 2> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>> ************* [junit4] 2> at
>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>> [junit4] 2> at
>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>> [junit4] 2> at
>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>> [junit4] 2> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>> [junit4] 2> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>> [junit4] 2> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>> [junit4] 2> at
>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>> [junit4] 2> at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>> [junit4] 2> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>> [junit4] 2> at
>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>> [junit4] 2> ... 23 more
>>
>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>> wrote:
>>> Looking......
>>>
>>>
>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>> <je...@thetaphi.de> wrote:
>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>
>>>> 4 tests failed.
>>>> FAILED:
>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>
>>>> Error Message:
>>>> Some resources were not closed, shutdown, or released.
>>>>
>>>> Stack Trace:
>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>> released.
>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>> at org.junit.Assert.fail(Assert.java:93)
>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>> at
>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>> at
>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>> at
>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>> at
>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>> at
>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>> at
>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>> at
>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> FAILED:
>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>
>>>> Error Message:
>>>> 5 threads leaked from SUITE scope at
>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>> Thread[id=4675,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> java.lang.Thread.sleep(Native Method) at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> Stack Trace:
>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>> 1) Thread[id=4675,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 3) Thread[id=4676,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>
>>>>
>>>> FAILED:
>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>
>>>> Error Message:
>>>> There are still zombie threads that couldn't be terminated: 1)
>>>> Thread[id=4675,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> java.lang.Thread.sleep(Native Method) at
>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.ut
>>
>> [The entire original message is not included.]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
everything just fine, my laptop fails all the time.
On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
<ic...@gmail.com> wrote:
> Not sure if it is related or helpful, but while debugging tests for
> SOLR-7468 yesterday, I encountered this
> java.lang.NoSuchFieldError:
> totalTermCount
>
> few times, I had to forcefully clean at root of the project and it worked. I
> remember Anshum had to do that clean thing more than once to make it work
> and he remarked "don't ask why".
>
> Sent from my Windows Phone
> ________________________________
> From: Erick Erickson
> Sent: 5/23/2015 6:15 AM
> To: dev@lucene.apache.org
> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
> #12782 - Failure!
>
> OK, this is somewhat weird. I still have the original tree that I
> checked in from which was up to date before I committed the code and
> the tests run from there fine. But a current trunk fails every time.
> Now, the machine it works on is my Mac Pro, and the failures are on my
> MacBook so there may be something going on there.
>
> I've got to leave for a while, I'll copy the tree that works on the
> Pro, update the copy and see if this test fails when I get back. If
> they fail, I can diff the trees to see what changed and see if I can
> make any sense out of this.
>
> I can always @Ignore this test to cut down on the noise, probably do
> that tonight if I don't have any revelations.
>
> I see this stack trace which makes no sense to me whatsoever (see the
> lines with lots of ***** in front). I looked at where the code
> originates (BufferedUpdatesStream[277]) and it looks like this:
>
> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>
> And it's telling me there's no such field? Whaaaaatttt????
>
> Which is freaking me out since I don't see how this would trigger the
> exception. Is this a red herring? And, of course, this doesn't fail in
> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>
> Of course if this were something fundamental to Lucene, it seems like
> this would be failing all over the place so I assume it's something to
> do with CDCR... But what do I know?
>
> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> status=0 QTime=8
> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
> c:source_collection s:shard1 r:core_node3
> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
> totalTermCount
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
> [junit4] 2> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> [junit4] 2> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> [junit4] 2> at
> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> [junit4] 2> at
> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
> [junit4] 2> at
> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> [junit4] 2> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> [junit4] 2> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
> [junit4] 2> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> [junit4] 2> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> [junit4] 2> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> [junit4] 2> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> [junit4] 2> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
> ************* [junit4] 2> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
> [junit4] 2> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
> [junit4] 2> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
> [junit4] 2> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
> [junit4] 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
> [junit4] 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
> [junit4] 2> at
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
> [junit4] 2> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> [junit4] 2> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> [junit4] 2> at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
> [junit4] 2> ... 23 more
>
> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
> wrote:
>> Looking......
>>
>>
>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>> <je...@thetaphi.de> wrote:
>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>
>>> 4 tests failed.
>>> FAILED:
>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>
>>> Error Message:
>>> Some resources were not closed, shutdown, or released.
>>>
>>> Stack Trace:
>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>> released.
>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>> at org.junit.Assert.fail(Assert.java:93)
>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>> at
>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>> at
>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>> at
>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>> at
>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>> at
>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> FAILED:
>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>
>>> Error Message:
>>> 5 threads leaked from SUITE scope at
>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>> Thread[id=4675,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>> java.lang.Thread.sleep(Native Method) at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Stack Trace:
>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>> 1) Thread[id=4675,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>> at java.lang.Thread.sleep(Native Method)
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> 3) Thread[id=4676,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> at
>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>
>>>
>>> FAILED:
>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>
>>> Error Message:
>>> There are still zombie threads that couldn't be terminated: 1)
>>> Thread[id=4675,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>> java.lang.Thread.sleep(Native Method) at
>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.ut
>
> [The entire original message is not included.]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org