You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Kevin Risden <co...@gmail.com> on 2016/10/06 23:02:13 UTC
Lucene/Solr Master Nightly Tests 1126 stuck?
https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1126/
Its been running for 1 day 19 hrs. Usually it runs in 5-8hours?
Kevin Risden
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Dawid Weiss <da...@gmail.com>.
Here's another twilight zone moment:
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.isAlive(Native Method)
at java.lang.Thread.join(Thread.java:1248)
- locked <0x00000000e016d798> (a
com.carrotsearch.ant.tasks.junit4.events.Serializer$1)
This thread is waiting for the shutdown deamon thread. It should.
Except that the daemon thread (the one being joined) is gone (not in
the thread dump) and the loop above spins (RUNNABLE state) on:
while (isAlive()) {
wait(0);
}
Another impossibility.
Dawid
On Thu, Oct 27, 2016 at 2:42 PM, Dawid Weiss <da...@gmail.com> wrote:
> Just like I thought. This isn't currently fixable without external
> termination of the forked process. The forked JVM ran out of memory
> and deadlocked.
>
> [junit4] WARN: Unhandled exception in event serialization. ->
> java.lang.OutOfMemoryError: Java heap space
> [junit4] WARN: Unreachable code. Complete panic. ->
> java.lang.OutOfMemoryError: Java heap space
>
> What's interesting about it, sadly, is the JVM should have quit, but
> didn't. How the OOM is re-thrown from the finally block in this code is
> a mystery to me, but like I said low-memory conditions are uncharted
> territory and there's very little one can "assume" about them from
> Java level.
>
> [...]
> if (reason != null) {
> try {
> SlaveMain.warn("Unhandled exception in event
> serialization.", reason);
> } finally {
> Runtime.getRuntime().halt(0);
> }
> }
> }
> } catch (Throwable t) {
> SlaveMain.warn("Unreachable code. Complete panic.", t);
> }
>
> Dawid
>
> On Wed, Oct 26, 2016 at 7:45 PM, Dawid Weiss <da...@gmail.com> wrote:
>> Uwe has collected stack traces, I will analyze later. Thanks for the
>> ping, Kevin.
>>
>> Dawid
>>
>> On Wed, Oct 26, 2016 at 7:32 PM, Dawid Weiss <da...@gmail.com> wrote:
>>> Uwe, can you try to take a jps (or send a signal) to the forked JVM
>>> (and the master)? Thanks!
>>>
>>> Dawid
>>>
>>> On Wed, Oct 26, 2016 at 5:06 PM, Kevin Risden <co...@gmail.com> wrote:
>>>> Looks like nightly 6.x is stalled in the same way.
>>>> https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/185
>>>>
>>>> Typically this takes ~4-6 hours and it is on 23+ and counting.
>>>>
>>>> Kevin Risden
>>>>
>>>> On Mon, Oct 10, 2016 at 12:08 PM, Dawid Weiss <da...@gmail.com> wrote:
>>>>>
>>>>> Thanks Uwe, this helps a lot!
>>>>>
>>>>> There is a resource deadlock here (an interplay of loggers, sysouts
>>>>> and junit4 stream redirectors and uncaught exception handlers...).
>>>>> It's really complex, but I'll try to get to the bottom of it.
>>>>>
>>>>> This completely aside, over 40 THOUSAND threads are hanging inside
>>>>> jetty's http handlers... there should be a more reasonable limit to
>>>>> this I guess?!
>>>>>
>>>>> "qtp1445698227-45502" #45502 prio=5 os_prio=0 tid=0x00007f5f5447c000
>>>>> nid=0x4ec1 waiting for monitor entry [0x00007f5f26327000]
>>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>>> at org.apache.log4j.Category.callAppenders(Category.java:204)
>>>>> - waiting to lock <0x00000000e00a8348> (a org.apache.log4j.spi.RootLogger)
>>>>> at org.apache.log4j.Category.forcedLog(Category.java:391)
>>>>> at org.apache.log4j.Category.log(Category.java:856)
>>>>> at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:497)
>>>>> at org.apache.solr.common.SolrException.log(SolrException.java:159)
>>>>> at
>>>>> org.apache.solr.servlet.ResponseUtils.getErrorInfo(ResponseUtils.java:65)
>>>>>
>>>>> Dawid
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Dawid Weiss <da...@gmail.com>.
Just like I thought. This isn't currently fixable without external
termination of the forked process. The forked JVM ran out of memory
and deadlocked.
[junit4] WARN: Unhandled exception in event serialization. ->
java.lang.OutOfMemoryError: Java heap space
[junit4] WARN: Unreachable code. Complete panic. ->
java.lang.OutOfMemoryError: Java heap space
What's interesting about it, sadly, is the JVM should have quit, but
didn't. How the OOM is re-thrown from the finally block in this code is
a mystery to me, but like I said low-memory conditions are uncharted
territory and there's very little one can "assume" about them from
Java level.
[...]
if (reason != null) {
try {
SlaveMain.warn("Unhandled exception in event
serialization.", reason);
} finally {
Runtime.getRuntime().halt(0);
}
}
}
} catch (Throwable t) {
SlaveMain.warn("Unreachable code. Complete panic.", t);
}
Dawid
On Wed, Oct 26, 2016 at 7:45 PM, Dawid Weiss <da...@gmail.com> wrote:
> Uwe has collected stack traces, I will analyze later. Thanks for the
> ping, Kevin.
>
> Dawid
>
> On Wed, Oct 26, 2016 at 7:32 PM, Dawid Weiss <da...@gmail.com> wrote:
>> Uwe, can you try to take a jps (or send a signal) to the forked JVM
>> (and the master)? Thanks!
>>
>> Dawid
>>
>> On Wed, Oct 26, 2016 at 5:06 PM, Kevin Risden <co...@gmail.com> wrote:
>>> Looks like nightly 6.x is stalled in the same way.
>>> https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/185
>>>
>>> Typically this takes ~4-6 hours and it is on 23+ and counting.
>>>
>>> Kevin Risden
>>>
>>> On Mon, Oct 10, 2016 at 12:08 PM, Dawid Weiss <da...@gmail.com> wrote:
>>>>
>>>> Thanks Uwe, this helps a lot!
>>>>
>>>> There is a resource deadlock here (an interplay of loggers, sysouts
>>>> and junit4 stream redirectors and uncaught exception handlers...).
>>>> It's really complex, but I'll try to get to the bottom of it.
>>>>
>>>> This completely aside, over 40 THOUSAND threads are hanging inside
>>>> jetty's http handlers... there should be a more reasonable limit to
>>>> this I guess?!
>>>>
>>>> "qtp1445698227-45502" #45502 prio=5 os_prio=0 tid=0x00007f5f5447c000
>>>> nid=0x4ec1 waiting for monitor entry [0x00007f5f26327000]
>>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at org.apache.log4j.Category.callAppenders(Category.java:204)
>>>> - waiting to lock <0x00000000e00a8348> (a org.apache.log4j.spi.RootLogger)
>>>> at org.apache.log4j.Category.forcedLog(Category.java:391)
>>>> at org.apache.log4j.Category.log(Category.java:856)
>>>> at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:497)
>>>> at org.apache.solr.common.SolrException.log(SolrException.java:159)
>>>> at
>>>> org.apache.solr.servlet.ResponseUtils.getErrorInfo(ResponseUtils.java:65)
>>>>
>>>> Dawid
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Dawid Weiss <da...@gmail.com>.
Uwe has collected stack traces, I will analyze later. Thanks for the
ping, Kevin.
Dawid
On Wed, Oct 26, 2016 at 7:32 PM, Dawid Weiss <da...@gmail.com> wrote:
> Uwe, can you try to take a jps (or send a signal) to the forked JVM
> (and the master)? Thanks!
>
> Dawid
>
> On Wed, Oct 26, 2016 at 5:06 PM, Kevin Risden <co...@gmail.com> wrote:
>> Looks like nightly 6.x is stalled in the same way.
>> https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/185
>>
>> Typically this takes ~4-6 hours and it is on 23+ and counting.
>>
>> Kevin Risden
>>
>> On Mon, Oct 10, 2016 at 12:08 PM, Dawid Weiss <da...@gmail.com> wrote:
>>>
>>> Thanks Uwe, this helps a lot!
>>>
>>> There is a resource deadlock here (an interplay of loggers, sysouts
>>> and junit4 stream redirectors and uncaught exception handlers...).
>>> It's really complex, but I'll try to get to the bottom of it.
>>>
>>> This completely aside, over 40 THOUSAND threads are hanging inside
>>> jetty's http handlers... there should be a more reasonable limit to
>>> this I guess?!
>>>
>>> "qtp1445698227-45502" #45502 prio=5 os_prio=0 tid=0x00007f5f5447c000
>>> nid=0x4ec1 waiting for monitor entry [0x00007f5f26327000]
>>> java.lang.Thread.State: BLOCKED (on object monitor)
>>> at org.apache.log4j.Category.callAppenders(Category.java:204)
>>> - waiting to lock <0x00000000e00a8348> (a org.apache.log4j.spi.RootLogger)
>>> at org.apache.log4j.Category.forcedLog(Category.java:391)
>>> at org.apache.log4j.Category.log(Category.java:856)
>>> at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:497)
>>> at org.apache.solr.common.SolrException.log(SolrException.java:159)
>>> at
>>> org.apache.solr.servlet.ResponseUtils.getErrorInfo(ResponseUtils.java:65)
>>>
>>> Dawid
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Dawid Weiss <da...@gmail.com>.
Uwe, can you try to take a jps (or send a signal) to the forked JVM
(and the master)? Thanks!
Dawid
On Wed, Oct 26, 2016 at 5:06 PM, Kevin Risden <co...@gmail.com> wrote:
> Looks like nightly 6.x is stalled in the same way.
> https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/185
>
> Typically this takes ~4-6 hours and it is on 23+ and counting.
>
> Kevin Risden
>
> On Mon, Oct 10, 2016 at 12:08 PM, Dawid Weiss <da...@gmail.com> wrote:
>>
>> Thanks Uwe, this helps a lot!
>>
>> There is a resource deadlock here (an interplay of loggers, sysouts
>> and junit4 stream redirectors and uncaught exception handlers...).
>> It's really complex, but I'll try to get to the bottom of it.
>>
>> This completely aside, over 40 THOUSAND threads are hanging inside
>> jetty's http handlers... there should be a more reasonable limit to
>> this I guess?!
>>
>> "qtp1445698227-45502" #45502 prio=5 os_prio=0 tid=0x00007f5f5447c000
>> nid=0x4ec1 waiting for monitor entry [0x00007f5f26327000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.log4j.Category.callAppenders(Category.java:204)
>> - waiting to lock <0x00000000e00a8348> (a org.apache.log4j.spi.RootLogger)
>> at org.apache.log4j.Category.forcedLog(Category.java:391)
>> at org.apache.log4j.Category.log(Category.java:856)
>> at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:497)
>> at org.apache.solr.common.SolrException.log(SolrException.java:159)
>> at
>> org.apache.solr.servlet.ResponseUtils.getErrorInfo(ResponseUtils.java:65)
>>
>> Dawid
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Kevin Risden <co...@gmail.com>.
Looks like nightly 6.x is stalled in the same way.
https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/185
Typically this takes ~4-6 hours and it is on 23+ and counting.
Kevin Risden
On Mon, Oct 10, 2016 at 12:08 PM, Dawid Weiss <da...@gmail.com> wrote:
> Thanks Uwe, this helps a lot!
>
> There is a resource deadlock here (an interplay of loggers, sysouts
> and junit4 stream redirectors and uncaught exception handlers...).
> It's really complex, but I'll try to get to the bottom of it.
>
> This completely aside, over 40 THOUSAND threads are hanging inside
> jetty's http handlers... there should be a more reasonable limit to
> this I guess?!
>
> "qtp1445698227-45502" #45502 prio=5 os_prio=0 tid=0x00007f5f5447c000
> nid=0x4ec1 waiting for monitor entry [0x00007f5f26327000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.log4j.Category.callAppenders(Category.java:204)
> - waiting to lock <0x00000000e00a8348> (a org.apache.log4j.spi.RootLogger)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:497)
> at org.apache.solr.common.SolrException.log(SolrException.java:159)
> at org.apache.solr.servlet.ResponseUtils.getErrorInfo(
> ResponseUtils.java:65)
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Dawid Weiss <da...@gmail.com>.
Thanks Uwe, this helps a lot!
There is a resource deadlock here (an interplay of loggers, sysouts
and junit4 stream redirectors and uncaught exception handlers...).
It's really complex, but I'll try to get to the bottom of it.
This completely aside, over 40 THOUSAND threads are hanging inside
jetty's http handlers... there should be a more reasonable limit to
this I guess?!
"qtp1445698227-45502" #45502 prio=5 os_prio=0 tid=0x00007f5f5447c000
nid=0x4ec1 waiting for monitor entry [0x00007f5f26327000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:204)
- waiting to lock <0x00000000e00a8348> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:497)
at org.apache.solr.common.SolrException.log(SolrException.java:159)
at org.apache.solr.servlet.ResponseUtils.getErrorInfo(ResponseUtils.java:65)
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
the trace is in the workspace:
https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/ws/
The trace file is 9 Megabytes, but I attached it in bzip2 format.
I am working on killing processes! There are multiple hung processes still there.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, October 10, 2016 6:45 PM
> To: dev@lucene.apache.org; Dawid Weiss <da...@gmail.com>
> Subject: Re: Lucene/Solr Master Nightly Tests 1126 stuck?
>
> I will look into this.
>
> Am 10. Oktober 2016 18:38:40 MESZ, schrieb Dawid Weiss
> <da...@gmail.com>:
> >Well, somebody with root access would have to try to jps the forked
> >process (pid 6761) and see if he or she can get the stacktrace.
> >
> >D.
> >
> >On Mon, Oct 10, 2016 at 4:29 PM, Kevin Risden
> ><co...@gmail.com> wrote:
> >> Looks like this build is still running?
> >>
> >> Kevin Risden
> >>
> >> On Fri, Oct 7, 2016 at 1:39 AM, Dawid Weiss <da...@gmail.com>
> >wrote:
> >>>
> >>> > I wonder why the build didn't timeout at 2 hours?
> >>>
> >>> It won't if the JVM died (and Runtime.halt() didn't stop the process
> >>> for whatever reason).
> >>>
> >>> > How can we kill it?
> >>>
> >>> It'd be good to try to get a stack (if possible), although I doubt
> >it will
> >>> be.
> >>>
> >>> D.
> >>>
> >>>
> >---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>
> >>
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
> Uwe Schindler
> H.-H.-Meier-Allee 63, 28213 Bremen
> http://www.thetaphi.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Uwe Schindler <uw...@thetaphi.de>.
I will look into this.
Am 10. Oktober 2016 18:38:40 MESZ, schrieb Dawid Weiss <da...@gmail.com>:
>Well, somebody with root access would have to try to jps the forked
>process (pid 6761) and see if he or she can get the stacktrace.
>
>D.
>
>On Mon, Oct 10, 2016 at 4:29 PM, Kevin Risden
><co...@gmail.com> wrote:
>> Looks like this build is still running?
>>
>> Kevin Risden
>>
>> On Fri, Oct 7, 2016 at 1:39 AM, Dawid Weiss <da...@gmail.com>
>wrote:
>>>
>>> > I wonder why the build didn't timeout at 2 hours?
>>>
>>> It won't if the JVM died (and Runtime.halt() didn't stop the process
>>> for whatever reason).
>>>
>>> > How can we kill it?
>>>
>>> It'd be good to try to get a stack (if possible), although I doubt
>it will
>>> be.
>>>
>>> D.
>>>
>>>
>---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>For additional commands, e-mail: dev-help@lucene.apache.org
--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Dawid Weiss <da...@gmail.com>.
Well, somebody with root access would have to try to jps the forked
process (pid 6761) and see if he or she can get the stacktrace.
D.
On Mon, Oct 10, 2016 at 4:29 PM, Kevin Risden <co...@gmail.com> wrote:
> Looks like this build is still running?
>
> Kevin Risden
>
> On Fri, Oct 7, 2016 at 1:39 AM, Dawid Weiss <da...@gmail.com> wrote:
>>
>> > I wonder why the build didn't timeout at 2 hours?
>>
>> It won't if the JVM died (and Runtime.halt() didn't stop the process
>> for whatever reason).
>>
>> > How can we kill it?
>>
>> It'd be good to try to get a stack (if possible), although I doubt it will
>> be.
>>
>> D.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Kevin Risden <co...@gmail.com>.
Looks like this build is still running?
Kevin Risden
On Fri, Oct 7, 2016 at 1:39 AM, Dawid Weiss <da...@gmail.com> wrote:
> > I wonder why the build didn't timeout at 2 hours?
>
> It won't if the JVM died (and Runtime.halt() didn't stop the process
> for whatever reason).
>
> > How can we kill it?
>
> It'd be good to try to get a stack (if possible), although I doubt it will
> be.
>
> D.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Dawid Weiss <da...@gmail.com>.
> I wonder why the build didn't timeout at 2 hours?
It won't if the JVM died (and Runtime.halt() didn't stop the process
for whatever reason).
> How can we kill it?
It'd be good to try to get a stack (if possible), although I doubt it will be.
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene/Solr Master Nightly Tests 1126 stuck?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Yah, looks stuck:
[junit4] HEARTBEAT J2 PID(6761@lucene1-us-west):
2016-10-06T23:41:14, stalled for 135957s at:
CdcrReplicationDistributedZkTest.testDeleteCreateSourceCollection
I wonder why the build didn't timeout at 2 hours?
How can we kill it?
Mike McCandless
http://blog.mikemccandless.com
On Thu, Oct 6, 2016 at 7:02 PM, Kevin Risden <co...@gmail.com> wrote:
> https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1126/
>
> Its been running for 1 day 19 hrs. Usually it runs in 5-8hours?
>
> Kevin Risden
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org