You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mikhail Khludnev <mk...@apache.org> on 2018/09/14 09:10:07 UTC
Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
Hello, Vadim.
My guess (and only guess) that bunch of updates coming into a shard causes
a heavy merge that blocks new updates in its' order. This can be verified
with logs or threaddump from the problematic node. The probable measures
are: try to shuffle updates to load other shards for a while and let
parallel merge to pack that shard. And just wait a little by increasing
timeout in jetty.
Let us know what you will encounter.
On Thu, Sep 13, 2018 at 3:54 PM Vadim Ivanov <
vadim.ivanov@spb.ntk-intourist.ru> wrote:
> Hi,
> I've put some more tests on the issue and managed to find out more details.
> Time out occurs when while long indexing some documents in the beginning is
> going to one shard and then for a long time (more than 120 sec) no data at
> all is going to that shard.
> Connection to that core, opened in the beginning of indexing, goes to idle
> timeout :( .
> If no data at all going to the shard during indexing - no timeout occurs on
> that shard.
> If Indexing finishes earlier than 120 sec - no timeout occurs on that
> shard.
> Unfortunately, in our use-case there are lot of long indexing up to 30
> minutes with uneven shard distribution of documents.
> Any suggestion how to mitigate issue?
> --
> BR
> Vadim Ivanov
>
>
> -----Original Message-----
> From: Вадим Иванов [mailto:vadim.ivanov@spb.ntk-intourist.ru]
> Sent: Wednesday, September 12, 2018 4:29 PM
> To: solr-user@lucene.apache.org
> Subject: Idle Timeout while DIH indexing and implicit sharding in 7.4
>
> Hello gurus,
> I am using solrCloud with DIH for indexing my data.
> Testing 7.4.0 with implicitly sharded collection I have noticed that any
> indexing
> longer then 2 minutes always failing with many timeout records in log
> coming
> from all replicas in collection.
>
> Such as:
> x:Mycol_s_0_replica_t40 RequestHandlerBase
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 120001/120000 ms
> null:java.io.IOException: java.util.concurrent.TimeoutException: Idle
> timeout expired: 120000/120000 ms
> at
>
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1075)
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> at
>
> org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWra
> pper.java:74)
> ...
> Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
> 120000/120000 ms
> at
> org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
> at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
> 201(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
> eduledThreadPoolExecutor.java:293)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 49)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 24)
> ... 1 more
> Suppressed: java.lang.Throwable: HttpInput failure
> at
> org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:821)
> at
>
> org.eclipse.jetty.server.HttpConnection$BlockingReadCallback.failed(HttpConn
> ection.java:649)
> at
> org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:134)
>
> Resulting indexing status:
> "statusMessages":{
> "Total Requests made to DataSource":"1",
> "Total Rows Fetched":"2828323",
> "Total Documents Processed":"2828323",
> "Total Documents Skipped":"0",
> "Full Dump Started":"2018-09-12 14:28:21",
> "":"Indexing completed. Added/Updated: 2828323 documents. Deleted 0
> documents.",
> "Committed":"2018-09-12 14:33:41",
> "Time taken":"0:5:19.507",
> "Full Import failed":"2018-09-12 14:33:41"}}
>
> Nevertheless all these documents seems indexed fine and searchable.
> If the same collection not sharded or sharded as " compositeId" indexing
> done without any errors.
> Type of replicas - nrt or tolg doesn't matter.
> Small Indexing (taking less than 2 minutes) run smoothly.
>
> Testing environment - 1 node, Collection with 6 shards, 1 replica for each
> shard
> Collection:
> /admin/collections?action=CREATE&name=Mycol
> &numShards=6
> &router.name=implicit
> &shards=s_0,s_1,s_2,s_3,s_4,s_5
> &router.field=sf_shard
> &collection.configName=Mycol
> &maxShardsPerNode=10
> &nrtReplicas=0&tlogReplicas=1
>
>
> I have never noticed such behavior before on my prod configuration (solr
> 6.3.0)
> Seems like bug in new version, but I could not find any jira on issue.
>
> Any ideas, please...
>
> --
> BR
> Vadim Ivanov
>
>
--
Sincerely yours
Mikhail Khludnev
RE: Idle Timeout while DIH indexing and implicit sharding in 7.4
Posted by "swapna.minnakanti@copart.com" <sw...@copart.com>.
I am facing same exact issue. We never had any issue with 6.5.1 when doing
full index (initial bulk load)
After upgrading to 7.5.0, getting below exception and indexing is taking a
very long time
2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_member_lots_a s:shard1
r:core_node3 x:c_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 300000/300000 ms
at
org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
at
org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:74)
at
org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100)
at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
at
org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
at
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
at
org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
Posted by Mikhail Khludnev <mk...@apache.org>.
Tracked https://issues.apache.org/jira/browse/SOLR-13735 patches are
welcome.
On Mon, Sep 2, 2019 at 12:39 PM Vadim Ivanov <
vadim.ivanov@spb.ntk-intourist.ru> wrote:
> Timeout causes DIH to finish with error message. So, If I check DIH
> response to be sure
> that DIH session have finished without any mistakes.... it causes some
> trouble :)
> I haven't check yet whether all records successfully imported to solr.
> Supposed that after timeout shard does not accept records from DIH. Am I
> wrong?
> --
> Vadim
>
>
> > -----Original Message-----
> > From: Mikhail Khludnev [mailto:mkhl@apache.org]
> > Sent: Monday, September 02, 2019 12:23 PM
> > To: Vadim Ivanov; solr-user
> > Subject: Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
> >
> > It seems like reasonable behavior. SolrWriter hangs while import is
> > running, it holds DistributedZkUpdateProcessor, which holds
> > SolrCmdDistributor, which keep client connection to shards open, which
> > causes timeout.
> > It might be worked around by supplying custom SolrWriter which finishes
> > UpdateProcessor's chain from time to time and recreate it. However, that
> > exception shouldn't cause any problem or it does?
> > Also, it's worth to track as a jira, or mentioned in the ticket regarding
> > adjusting DIH for Cloud.
> >
> > On Mon, Sep 2, 2019 at 9:44 AM Vadim Ivanov <
> > vadim.ivanov@spb.ntk-intourist.ru> wrote:
> >
> > > I’ve raised that timeout from 5 min to 40 min.
> > >
> > > It somehow mitigated the issue in my use case.
> > >
> > > Problem occurs when some updates goes to one shard in the beginning of
> > > long DIH session, then all updates goes to other shards for more than
> 300
> > > sec.
> > >
> > > And when ( after 300 sec) some updates goes again to the first shard it
> > > faces closed connection problem.
> > >
> > >
> > >
> > > Either it’s right behavior or not – it’s hard to say….
> > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Vadim
> > >
> > > *From:* Mikhail Khludnev [mailto:mkhl@apache.org]
> > > *Sent:* Monday, September 02, 2019 1:31 AM
> > > *To:* solr-user
> > > *Cc:* vadim.ivanov@spb.ntk-intourist.ru
> > > *Subject:* Re: Idle Timeout while DIH indexing and implicit sharding in
> > > 7.4
> > >
> > >
> > >
> > > Giving that
> > >
> > >
> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> > > at
> > >
> > >
> > org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoad
> > er.java:107)
> > >
> > >
> > >
> > >
> > > JavabinLoader hangs on Stream.peek(), awaiting -1, and hit timeout. I
> > > guess it's might be related with "closing sockets". I've looked through
> > > CHANGES after 6.5.1, here's what might be examined for impacting this:
> > >
> > >
> > >
> > > * SOLR-10779: JavaBinCodec should use close consistently
> > >
> > > * SOLR-12290: Do not close any servlet streams and improve our servlet
> > > stream
> > >
> > > Description Resource Path Location
> > > * SOLR-12477: An update would return a client error(400) if it hit a
> > > AlreadyClos
> > >
> > > * SOLR-12897: Introduce AlreadyClosedException to clean up silly close
> /
> > > shutd
> > >
> > >
> > >
> > > Have nothing more than that so far.
> > >
> > >
> > >
> > > On Sun, Sep 1, 2019 at 8:50 PM swapna.minnakanti@copart.com <
> > > swapna.minnakanti@copart.com> wrote:
> > >
> > > I am facing same exact issue. We never had any issue with 6.5.1 when
> doing
> > > full index (initial bulk load)
> > > After upgrading to 7.5.0, getting below exception and indexing is
> taking a
> > > very long time
> > >
> > > 2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_collection
> s:shard1
> > > r:core_node3 x:c_collection_shard1_replica_n1]
> o.a.s.h.RequestHandlerBase
> > > java.io.IOException: java.util.concurrent.TimeoutException: Idle
> timeout
> > > expired: 300000/300000 ms
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
> > > at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> > > at
> > >
> > >
> > org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStream
> > Wrapper.java:74)
> > > at
> > >
> > >
> > org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:
> > 100)
> > > at
> > >
> > >
> > org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputS
> > tream.java:79)
> > > at
> > >
> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
> > > at
> > >
> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> > > at
> > >
> > >
> > org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoad
> > er.java:107)
> > > at
> > >
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> > > at
> > >
> > >
> > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandl
> > er.java:97)
> > > at
> > >
> > >
> > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Cont
> > entStreamHandlerBase.java:68)
> > > at
> > >
> > >
> > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandler
> > Base.java:199)
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> > > at
> > > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> > > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> > > at
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> > > at
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.j
> > ava:1634)
> > > at
> > >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:14
> > 6)
> > > at
> > >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java
> > :132)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.jav
> > a:257)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:
> > 1595)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.jav
> > a:255)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.jav
> > a:1317)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java
> > :203)
> > > at
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1
> > 564)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java
> > :201)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:
> > 1219)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:14
> > 4)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHan
> > dlerCollection.java:219)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.ja
> > va:126)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java
> > :132)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:
> > 335)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java
> > :132)
> > > at org.eclipse.jetty.server.Server.handle(Server.java:531)
> > > at
> > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
> > > at
> > >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
> > > at
> > > org.eclipse.jetty.io
> > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
> > > at org.eclipse.jetty.io
> > > .FillInterest.fillable(FillInterest.java:102)
> > > at
> > > org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
> > > at
> > >
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.j
> > ava:333)
> > >
> > >
> > >
> > > --
> > > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
>
>
--
Sincerely yours
Mikhail Khludnev
RE: Idle Timeout while DIH indexing and implicit sharding in 7.4
Posted by Vadim Ivanov <va...@spb.ntk-intourist.ru>.
Timeout causes DIH to finish with error message. So, If I check DIH response to be sure
that DIH session have finished without any mistakes.... it causes some trouble :)
I haven't check yet whether all records successfully imported to solr.
Supposed that after timeout shard does not accept records from DIH. Am I wrong?
--
Vadim
> -----Original Message-----
> From: Mikhail Khludnev [mailto:mkhl@apache.org]
> Sent: Monday, September 02, 2019 12:23 PM
> To: Vadim Ivanov; solr-user
> Subject: Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
>
> It seems like reasonable behavior. SolrWriter hangs while import is
> running, it holds DistributedZkUpdateProcessor, which holds
> SolrCmdDistributor, which keep client connection to shards open, which
> causes timeout.
> It might be worked around by supplying custom SolrWriter which finishes
> UpdateProcessor's chain from time to time and recreate it. However, that
> exception shouldn't cause any problem or it does?
> Also, it's worth to track as a jira, or mentioned in the ticket regarding
> adjusting DIH for Cloud.
>
> On Mon, Sep 2, 2019 at 9:44 AM Vadim Ivanov <
> vadim.ivanov@spb.ntk-intourist.ru> wrote:
>
> > I’ve raised that timeout from 5 min to 40 min.
> >
> > It somehow mitigated the issue in my use case.
> >
> > Problem occurs when some updates goes to one shard in the beginning of
> > long DIH session, then all updates goes to other shards for more than 300
> > sec.
> >
> > And when ( after 300 sec) some updates goes again to the first shard it
> > faces closed connection problem.
> >
> >
> >
> > Either it’s right behavior or not – it’s hard to say….
> >
> >
> >
> >
> >
> > --
> >
> > Vadim
> >
> > *From:* Mikhail Khludnev [mailto:mkhl@apache.org]
> > *Sent:* Monday, September 02, 2019 1:31 AM
> > *To:* solr-user
> > *Cc:* vadim.ivanov@spb.ntk-intourist.ru
> > *Subject:* Re: Idle Timeout while DIH indexing and implicit sharding in
> > 7.4
> >
> >
> >
> > Giving that
> >
> > org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> > at
> >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoad
> er.java:107)
> >
> >
> >
> >
> > JavabinLoader hangs on Stream.peek(), awaiting -1, and hit timeout. I
> > guess it's might be related with "closing sockets". I've looked through
> > CHANGES after 6.5.1, here's what might be examined for impacting this:
> >
> >
> >
> > * SOLR-10779: JavaBinCodec should use close consistently
> >
> > * SOLR-12290: Do not close any servlet streams and improve our servlet
> > stream
> >
> > Description Resource Path Location
> > * SOLR-12477: An update would return a client error(400) if it hit a
> > AlreadyClos
> >
> > * SOLR-12897: Introduce AlreadyClosedException to clean up silly close /
> > shutd
> >
> >
> >
> > Have nothing more than that so far.
> >
> >
> >
> > On Sun, Sep 1, 2019 at 8:50 PM swapna.minnakanti@copart.com <
> > swapna.minnakanti@copart.com> wrote:
> >
> > I am facing same exact issue. We never had any issue with 6.5.1 when doing
> > full index (initial bulk load)
> > After upgrading to 7.5.0, getting below exception and indexing is taking a
> > very long time
> >
> > 2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_collection s:shard1
> > r:core_node3 x:c_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase
> > java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> > expired: 300000/300000 ms
> > at
> >
> >
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
> > at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> > at
> >
> >
> org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStream
> Wrapper.java:74)
> > at
> >
> >
> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:
> 100)
> > at
> >
> >
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputS
> tream.java:79)
> > at
> > org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
> > at
> > org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> > at
> >
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoad
> er.java:107)
> > at
> > org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> > at
> >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandl
> er.java:97)
> > at
> >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Cont
> entStreamHandlerBase.java:68)
> > at
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandler
> Base.java:199)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> > at
> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> > at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> > at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> > at
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.j
> ava:1634)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:14
> 6)
> > at
> > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java
> :132)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.jav
> a:257)
> > at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:
> 1595)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.jav
> a:255)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.jav
> a:1317)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java
> :203)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> > at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1
> 564)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java
> :201)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:
> 1219)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:14
> 4)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHan
> dlerCollection.java:219)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.ja
> va:126)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java
> :132)
> > at
> >
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:
> 335)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java
> :132)
> > at org.eclipse.jetty.server.Server.handle(Server.java:531)
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
> > at
> > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
> > at
> > org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
> > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:102)
> > at
> > org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
> > at
> >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.j
> ava:333)
> >
> >
> >
> > --
> > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >
> >
> >
> >
> > --
> >
> > Sincerely yours
> > Mikhail Khludnev
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
Posted by Mikhail Khludnev <mk...@apache.org>.
It seems like reasonable behavior. SolrWriter hangs while import is
running, it holds DistributedZkUpdateProcessor, which holds
SolrCmdDistributor, which keep client connection to shards open, which
causes timeout.
It might be worked around by supplying custom SolrWriter which finishes
UpdateProcessor's chain from time to time and recreate it. However, that
exception shouldn't cause any problem or it does?
Also, it's worth to track as a jira, or mentioned in the ticket regarding
adjusting DIH for Cloud.
On Mon, Sep 2, 2019 at 9:44 AM Vadim Ivanov <
vadim.ivanov@spb.ntk-intourist.ru> wrote:
> I’ve raised that timeout from 5 min to 40 min.
>
> It somehow mitigated the issue in my use case.
>
> Problem occurs when some updates goes to one shard in the beginning of
> long DIH session, then all updates goes to other shards for more than 300
> sec.
>
> And when ( after 300 sec) some updates goes again to the first shard it
> faces closed connection problem.
>
>
>
> Either it’s right behavior or not – it’s hard to say….
>
>
>
>
>
> --
>
> Vadim
>
> *From:* Mikhail Khludnev [mailto:mkhl@apache.org]
> *Sent:* Monday, September 02, 2019 1:31 AM
> *To:* solr-user
> *Cc:* vadim.ivanov@spb.ntk-intourist.ru
> *Subject:* Re: Idle Timeout while DIH indexing and implicit sharding in
> 7.4
>
>
>
> Giving that
>
> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> at
>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
>
>
>
>
> JavabinLoader hangs on Stream.peek(), awaiting -1, and hit timeout. I
> guess it's might be related with "closing sockets". I've looked through
> CHANGES after 6.5.1, here's what might be examined for impacting this:
>
>
>
> * SOLR-10779: JavaBinCodec should use close consistently
>
> * SOLR-12290: Do not close any servlet streams and improve our servlet
> stream
>
> Description Resource Path Location
> * SOLR-12477: An update would return a client error(400) if it hit a
> AlreadyClos
>
> * SOLR-12897: Introduce AlreadyClosedException to clean up silly close /
> shutd
>
>
>
> Have nothing more than that so far.
>
>
>
> On Sun, Sep 1, 2019 at 8:50 PM swapna.minnakanti@copart.com <
> swapna.minnakanti@copart.com> wrote:
>
> I am facing same exact issue. We never had any issue with 6.5.1 when doing
> full index (initial bulk load)
> After upgrading to 7.5.0, getting below exception and indexing is taking a
> very long time
>
> 2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_collection s:shard1
> r:core_node3 x:c_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 300000/300000 ms
> at
>
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> at
>
> org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:74)
> at
>
> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100)
> at
>
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
> at
> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
> at
> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> at
>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
> at
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> at
>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
> at
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
> at
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
>
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:531)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
> at
> org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
> at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:102)
> at
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
> at
>
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
>
>
>
> --
>
> Sincerely yours
> Mikhail Khludnev
>
--
Sincerely yours
Mikhail Khludnev
Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
Posted by Mikhail Khludnev <mk...@apache.org>.
Giving that
org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
at
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
s(JavabinLoader.java:107)
JavabinLoader hangs on Stream.peek(), awaiting -1, and hit timeout. I guess
it's might be related with "closing sockets". I've looked through CHANGES
after 6.5.1, here's what might be examined for impacting this:
* SOLR-10779: JavaBinCodec should use close consistently
* SOLR-12290: Do not close any servlet streams and improve our servlet
stream
Description Resource Path Location
* SOLR-12477: An update would return a client error(400) if it hit a
AlreadyClos
* SOLR-12897: Introduce AlreadyClosedException to clean up silly close /
shutd
Have nothing more than that so far.
On Sun, Sep 1, 2019 at 8:50 PM swapna.minnakanti@copart.com <
swapna.minnakanti@copart.com> wrote:
> I am facing same exact issue. We never had any issue with 6.5.1 when doing
> full index (initial bulk load)
> After upgrading to 7.5.0, getting below exception and indexing is taking a
> very long time
>
> 2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_collection s:shard1
> r:core_node3 x:c_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 300000/300000 ms
> at
>
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> at
>
> org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:74)
> at
>
> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100)
> at
>
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
> at
> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
> at
> org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
> at
>
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
> at
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> at
>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
> at
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
> at
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
>
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:531)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
> at
> org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
> at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:102)
> at
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
> at
>
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
--
Sincerely yours
Mikhail Khludnev
RE: Idle Timeout while DIH indexing and implicit sharding in 7.4
Posted by "swapna.minnakanti@copart.com" <sw...@copart.com>.
I am facing same exact issue. We never had any issue with 6.5.1 when doing
full index (initial bulk load)
After upgrading to 7.5.0, getting below exception and indexing is taking a
very long time
2019-09-01 10:11:27.436 ERROR (qtp1650813924-22) [c:c_collection s:shard1
r:core_node3 x:c_collection_shard1_replica_n1] o.a.s.h.RequestHandlerBase
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 300000/300000 ms
at
org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1080)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
at
org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:74)
at
org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:100)
at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
at
org.apache.solr.common.util.FastInputStream.peek(FastInputStream.java:60)
at
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
at
org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
RE: Idle Timeout while DIH indexing and implicit sharding in 7.4
Posted by Vadim Ivanov <va...@spb.ntk-intourist.ru>.
Hello
Mikhail, thank you for support.
I have already tested this case a lot to be sure what is happening under the hood.
As you proposed - I 've shuffled data coming from sql to solr to see how solr reacts:
I have 6 shards s0 ... s5
shard - is the routing field in my collection. (router.name=implicit&router.field=shard)
Му sql query looks like this
Select
id
....
, Case when 100 > RowNumber then 's5'
else 's_' + cast(RowNumber % 4 as varchar)
end as shard
from ...
Оnly first 99 rows goes to shard s5, and all the rest data spreads evenly between s0 ... s3.
After 120 sec of indexing I receive IdleTimeout from shard leader of s5
s4 receives no data and seems do not open connection at all - so no Timeout occurs
s0...s3 receives data and no Timeout occurs
When I tweak IdleTimeout in /opt/solr-7.4.0/server/etc/jetty-http.xml It helps,
But I have concerns about icreasing it from 120sec to 30 min.
Is it safe? What consequences could be?
I have noticed that IdleTimeout in jetty 9.3.8 (coming with Solr 6.3.0) was 50 sec
And no such behavior was observed in Solr 6.3. So default was increased significantly in 9.4.10 for some reason.
Maybe someone could shed some light on the reasons. What was changed in document routing behavior and why?
Maybe there was discussion about it that I could not find?
--
BR Vadim
-----Original Message-----
From: Mikhail Khludnev [mailto:mkhl@apache.org]
Sent: Friday, September 14, 2018 12:10 PM
To: solr-user
Subject: Re: Idle Timeout while DIH indexing and implicit sharding in 7.4
Hello, Vadim.
My guess (and only guess) that bunch of updates coming into a shard causes
a heavy merge that blocks new updates in its' order. This can be verified
with logs or threaddump from the problematic node. The probable measures
are: try to shuffle updates to load other shards for a while and let
parallel merge to pack that shard. And just wait a little by increasing
timeout in jetty.
Let us know what you will encounter.
On Thu, Sep 13, 2018 at 3:54 PM Vadim Ivanov <
vadim.ivanov@spb.ntk-intourist.ru> wrote:
> Hi,
> I've put some more tests on the issue and managed to find out more details.
> Time out occurs when while long indexing some documents in the beginning is
> going to one shard and then for a long time (more than 120 sec) no data at
> all is going to that shard.
> Connection to that core, opened in the beginning of indexing, goes to idle
> timeout :( .
> If no data at all going to the shard during indexing - no timeout occurs on
> that shard.
> If Indexing finishes earlier than 120 sec - no timeout occurs on that
> shard.
> Unfortunately, in our use-case there are lot of long indexing up to 30
> minutes with uneven shard distribution of documents.
> Any suggestion how to mitigate issue?
> --
> BR
> Vadim Ivanov
>
>
> -----Original Message-----
> From: Вадим Иванов [mailto:vadim.ivanov@spb.ntk-intourist.ru]
> Sent: Wednesday, September 12, 2018 4:29 PM
> To: solr-user@lucene.apache.org
> Subject: Idle Timeout while DIH indexing and implicit sharding in 7.4
>
> Hello gurus,
> I am using solrCloud with DIH for indexing my data.
> Testing 7.4.0 with implicitly sharded collection I have noticed that any
> indexing
> longer then 2 minutes always failing with many timeout records in log
> coming
> from all replicas in collection.
>
> Such as:
> x:Mycol_s_0_replica_t40 RequestHandlerBase
> java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
> expired: 120001/120000 ms
> null:java.io.IOException: java.util.concurrent.TimeoutException: Idle
> timeout expired: 120000/120000 ms
> at
>
> org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1075)
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:313)
> at
>
> org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWra
> pper.java:74)
> ...
> Caused by: java.util.concurrent.TimeoutException: Idle timeout expired:
> 120000/120000 ms
> at
> org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
> at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
> 201(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
> eduledThreadPoolExecutor.java:293)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 49)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 24)
> ... 1 more
> Suppressed: java.lang.Throwable: HttpInput failure
> at
> org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:821)
> at
>
> org.eclipse.jetty.server.HttpConnection$BlockingReadCallback.failed(HttpConn
> ection.java:649)
> at
> org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:134)
>
> Resulting indexing status:
> "statusMessages":{
> "Total Requests made to DataSource":"1",
> "Total Rows Fetched":"2828323",
> "Total Documents Processed":"2828323",
> "Total Documents Skipped":"0",
> "Full Dump Started":"2018-09-12 14:28:21",
> "":"Indexing completed. Added/Updated: 2828323 documents. Deleted 0
> documents.",
> "Committed":"2018-09-12 14:33:41",
> "Time taken":"0:5:19.507",
> "Full Import failed":"2018-09-12 14:33:41"}}
>
> Nevertheless all these documents seems indexed fine and searchable.
> If the same collection not sharded or sharded as " compositeId" indexing
> done without any errors.
> Type of replicas - nrt or tolg doesn't matter.
> Small Indexing (taking less than 2 minutes) run smoothly.
>
> Testing environment - 1 node, Collection with 6 shards, 1 replica for each
> shard
> Collection:
> /admin/collections?action=CREATE&name=Mycol
> &numShards=6
> &router.name=implicit
> &shards=s_0,s_1,s_2,s_3,s_4,s_5
> &router.field=sf_shard
> &collection.configName=Mycol
> &maxShardsPerNode=10
> &nrtReplicas=0&tlogReplicas=1
>
>
> I have never noticed such behavior before on my prod configuration (solr
> 6.3.0)
> Seems like bug in new version, but I could not find any jira on issue.
>
> Any ideas, please...
>
> --
> BR
> Vadim Ivanov
>
>
--
Sincerely yours
Mikhail Khludnev