You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Joe Obernberger <jo...@gmail.com> on 2018/04/02 19:55:28 UTC
Solr 7.1.0 - concurrent.ExecutionException building model
Hi All - when building machine learning models using information gain, I
sometimes get this error when the number of iterations is high. I'm
using about 20k news articles in my training set (about 10k positive,
and 10k negative), and (for this particular run) am using 500 terms and
25,000 iterations. I have gotten the error with a much lower number of
iterations (1,000) as well.
The specific stream command was:
update(models,
batchSize="50",train(MODEL1024_1522696624083,features(MODEL1024_1522696624083,q="*:*",featureSet="FSet_MODEL1024_1522696624083",field="Text",outcome="out_i",positiveLabel=1,numTerms=500),q="*:*",name="MODEL1024",field="Text",outcome="out_i",maxIterations="25000"))
The training data was split across 20 shards - specifically created with:
http://icarus.querymasters.com:9100/solr/admin/collections?action=CREATE&name=MODEL1024_1522696624083&numShards=20&replicationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
Any ideas? The complete error is:
java.io.IOException: java.util.concurrent.ExecutionException:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at
http://vesta:9100/solr/MODEL1024_1522696624083_shard20_replica_n75:
Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing
/solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
<pre> Not Found</pre></p>
</body>
</html>
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.read(TextLogitStream.java:498)
at
org.apache.solr.client.solrj.io.stream.PushBackStream.read(PushBackStream.java:87)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.read(UpdateStream.java:109)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:68)
at
org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:627)
at
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:87)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:84)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at
http://vesta:9100/solr/MODEL1024_1522696624083_shard20_replica_n75:
Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing
/solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
<pre> Not Found</pre></p>
</body>
</html>
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.read(TextLogitStream.java:459)
... 47 more
Caused by:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at
http://vesta:9100/solr/MODEL1024_1522696624083_shard20_replica_n75:
Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing
/solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
<pre> Not Found</pre></p>
</body>
</html>
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at
org.apache.solr.client.solrj.io.stream.TextLogitStream$LogitCall.call(TextLogitStream.java:640)
at
org.apache.solr.client.solrj.io.stream.TextLogitStream$LogitCall.call(TextLogitStream.java:582)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Last Check: 4/2/2018, 3:47:15 PM
Thank you!
-Joe Obernberger
Re: Solr 7.1.0 - concurrent.ExecutionException building model
Posted by Joe Obernberger <jo...@gmail.com>.
Hi Joel - thank you for your reply. Yes, the machine (Vesta) is up, and
I can access it. I don't see anything specific in the log, apart from
the same error, but this time to a different server. We have constant
indexing happening on this cluster, so if one went down, the indexing
would stop, and I've not seen that happen.
Interestingly, despite the error, the model is still built at least up
to some number of iterations. In other words, many iterations complete OK.
-Joe
On 4/2/2018 6:54 PM, Joel Bernstein wrote:
> It looks like it accessing a replica that's down. Are the logs from
> http://vesta:9100/solr/MODEL1024_1522696624083_shard20_replica_n75 reporting
> any issues? When you go to that url is it back up and running?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Mon, Apr 2, 2018 at 3:55 PM, Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> Hi All - when building machine learning models using information gain, I
>> sometimes get this error when the number of iterations is high. I'm using
>> about 20k news articles in my training set (about 10k positive, and 10k
>> negative), and (for this particular run) am using 500 terms and 25,000
>> iterations. I have gotten the error with a much lower number of iterations
>> (1,000) as well.
>>
>> The specific stream command was:
>> update(models, batchSize="50",train(MODEL1024_1522696624083,features(
>> MODEL1024_1522696624083,q="*:*",featureSet="FSet_MODEL1024_1
>> 522696624083",field="Text",outcome="out_i",positiveLabel=1,
>> numTerms=500),q="*:*",name="MODEL1024",field="Text",outcome=
>> "out_i",maxIterations="25000"))
>>
>> The training data was split across 20 shards - specifically created with:
>> http://icarus.querymasters.com:9100/solr/admin/collections?
>> action=CREATE&name=MODEL1024_1522696624083&numShards=20&rep
>> licationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
>>
>> Any ideas? The complete error is:
>>
>> java.io.IOException: java.util.concurrent.ExecutionException:
>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> Error from server at http://vesta:9100/solr/MODEL10
>> 24_1522696624083_shard20_replica_n75: Expected mime type
>> application/octet-stream but got text/html. <html>
>> <head>
>> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
>> <title>Error 404 Not Found</title>
>> </head>
>> <body><h2>HTTP ERROR 404</h2>
>> <p>Problem accessing /solr/MODEL1024_1522696624083_shard20_replica_n75/select.
>> Reason:
>> <pre> Not Found</pre></p>
>> </body>
>> </html>
>>
>> at org.apache.solr.client.solrj.io.stream.TextLogitStream.read(
>> TextLogitStream.java:498)
>> at org.apache.solr.client.solrj.io.stream.PushBackStream.read(P
>> ushBackStream.java:87)
>> at org.apache.solr.client.solrj.io.stream.UpdateStream.read(Upd
>> ateStream.java:109)
>> at org.apache.solr.client.solrj.io.stream.ExceptionStream.read(
>> ExceptionStream.java:68)
>> at org.apache.solr.handler.StreamHandler$TimerStream.read(
>> StreamHandler.java:627)
>> at org.apache.solr.client.solrj.io.stream.TupleStream.lambda$wr
>> iteMap$0(TupleStream.java:87)
>> at org.apache.solr.response.JSONWriter.writeIterator(JSONRespon
>> seWriter.java:523)
>> at org.apache.solr.response.TextResponseWriter.writeVal(TextRes
>> ponseWriter.java:180)
>> at org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter
>> .java:559)
>> at org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(
>> TupleStream.java:84)
>> at org.apache.solr.response.JSONWriter.writeMap(JSONResponseWri
>> ter.java:547)
>> at org.apache.solr.response.TextResponseWriter.writeVal(TextRes
>> ponseWriter.java:198)
>> at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithD
>> ups(JSONResponseWriter.java:209)
>> at org.apache.solr.response.JSONWriter.writeNamedList(JSONRespo
>> nseWriter.java:325)
>> at org.apache.solr.response.JSONWriter.writeResponse(JSONRespon
>> seWriter.java:120)
>> at org.apache.solr.response.JSONResponseWriter.write(JSONRespon
>> seWriter.java:71)
>> at org.apache.solr.response.QueryResponseWriterUtil.writeQueryR
>> esponse(QueryResponseWriterUtil.java:65)
>> at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrC
>> all.java:806)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:382)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:326)
>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>> r(ServletHandler.java:1751)
>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>> dler.java:582)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:143)
>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
>> ndler.java:548)
>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(
>> SessionHandler.java:226)
>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>> ContextHandler.java:1180)
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>> ler.java:512)
>> at org.eclipse.jetty.server.session.SessionHandler.doScope(
>> SessionHandler.java:185)
>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(
>> ContextHandler.java:1112)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:141)
>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
>> ndle(ContextHandlerCollection.java:213)
>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(
>> HandlerCollection.java:119)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> erWrapper.java:134)
>> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
>> iteHandler.java:335)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> erWrapper.java:134)
>> at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>> ction.java:251)
>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
>> succeeded(AbstractConnection.java:283)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
>> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
>> elEndPoint.java:93)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .executeProduceConsume(ExecuteProduceConsume.java:303)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .produceConsume(ExecuteProduceConsume.java:148)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .run(ExecuteProduceConsume.java:136)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>> ThreadPool.java:671)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
>> hreadPool.java:589)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.util.concurrent.ExecutionException:
>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> Error from server at http://vesta:9100/solr/MODEL10
>> 24_1522696624083_shard20_replica_n75: Expected mime type
>> application/octet-stream but got text/html. <html>
>> <head>
>> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
>> <title>Error 404 Not Found</title>
>> </head>
>> <body><h2>HTTP ERROR 404</h2>
>> <p>Problem accessing /solr/MODEL1024_1522696624083_shard20_replica_n75/select.
>> Reason:
>> <pre> Not Found</pre></p>
>> </body>
>> </html>
>>
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> at org.apache.solr.client.solrj.io.stream.TextLogitStream.read(
>> TextLogitStream.java:459)
>> ... 47 more
>> Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> Error from server at http://vesta:9100/solr/MODEL10
>> 24_1522696624083_shard20_replica_n75: Expected mime type
>> application/octet-stream but got text/html. <html>
>> <head>
>> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
>> <title>Error 404 Not Found</title>
>> </head>
>> <body><h2>HTTP ERROR 404</h2>
>> <p>Problem accessing /solr/MODEL1024_1522696624083_shard20_replica_n75/select.
>> Reason:
>> <pre> Not Found</pre></p>
>> </body>
>> </html>
>>
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
>> od(HttpSolrClient.java:590)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>> pSolrClient.java:253)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>> pSolrClient.java:242)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest
>> .java:178)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest
>> .java:195)
>> at org.apache.solr.client.solrj.io.stream.TextLogitStream$Logit
>> Call.call(TextLogitStream.java:640)
>> at org.apache.solr.client.solrj.io.stream.TextLogitStream$Logit
>> Call.call(TextLogitStream.java:582)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
>> xecutor.lambda$execute$0(ExecutorUtil.java:188)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:624)
>> ... 1 more
>> Last Check: 4/2/2018, 3:47:15 PM
>>
>> Thank you!
>>
>> -Joe Obernberger
>>
>>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>
Re: Solr 7.1.0 - concurrent.ExecutionException building model
Posted by Joel Bernstein <jo...@gmail.com>.
It looks like it accessing a replica that's down. Are the logs from
http://vesta:9100/solr/MODEL1024_1522696624083_shard20_replica_n75 reporting
any issues? When you go to that url is it back up and running?
Joel Bernstein
http://joelsolr.blogspot.com/
On Mon, Apr 2, 2018 at 3:55 PM, Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Hi All - when building machine learning models using information gain, I
> sometimes get this error when the number of iterations is high. I'm using
> about 20k news articles in my training set (about 10k positive, and 10k
> negative), and (for this particular run) am using 500 terms and 25,000
> iterations. I have gotten the error with a much lower number of iterations
> (1,000) as well.
>
> The specific stream command was:
> update(models, batchSize="50",train(MODEL1024_1522696624083,features(
> MODEL1024_1522696624083,q="*:*",featureSet="FSet_MODEL1024_1
> 522696624083",field="Text",outcome="out_i",positiveLabel=1,
> numTerms=500),q="*:*",name="MODEL1024",field="Text",outcome=
> "out_i",maxIterations="25000"))
>
> The training data was split across 20 shards - specifically created with:
> http://icarus.querymasters.com:9100/solr/admin/collections?
> action=CREATE&name=MODEL1024_1522696624083&numShards=20&rep
> licationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
>
> Any ideas? The complete error is:
>
> java.io.IOException: java.util.concurrent.ExecutionException:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://vesta:9100/solr/MODEL10
> 24_1522696624083_shard20_replica_n75: Expected mime type
> application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 404 Not Found</title>
> </head>
> <body><h2>HTTP ERROR 404</h2>
> <p>Problem accessing /solr/MODEL1024_1522696624083_shard20_replica_n75/select.
> Reason:
> <pre> Not Found</pre></p>
> </body>
> </html>
>
> at org.apache.solr.client.solrj.io.stream.TextLogitStream.read(
> TextLogitStream.java:498)
> at org.apache.solr.client.solrj.io.stream.PushBackStream.read(P
> ushBackStream.java:87)
> at org.apache.solr.client.solrj.io.stream.UpdateStream.read(Upd
> ateStream.java:109)
> at org.apache.solr.client.solrj.io.stream.ExceptionStream.read(
> ExceptionStream.java:68)
> at org.apache.solr.handler.StreamHandler$TimerStream.read(
> StreamHandler.java:627)
> at org.apache.solr.client.solrj.io.stream.TupleStream.lambda$wr
> iteMap$0(TupleStream.java:87)
> at org.apache.solr.response.JSONWriter.writeIterator(JSONRespon
> seWriter.java:523)
> at org.apache.solr.response.TextResponseWriter.writeVal(TextRes
> ponseWriter.java:180)
> at org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter
> .java:559)
> at org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(
> TupleStream.java:84)
> at org.apache.solr.response.JSONWriter.writeMap(JSONResponseWri
> ter.java:547)
> at org.apache.solr.response.TextResponseWriter.writeVal(TextRes
> ponseWriter.java:198)
> at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithD
> ups(JSONResponseWriter.java:209)
> at org.apache.solr.response.JSONWriter.writeNamedList(JSONRespo
> nseWriter.java:325)
> at org.apache.solr.response.JSONWriter.writeResponse(JSONRespon
> seWriter.java:120)
> at org.apache.solr.response.JSONResponseWriter.write(JSONRespon
> seWriter.java:71)
> at org.apache.solr.response.QueryResponseWriterUtil.writeQueryR
> esponse(QueryResponseWriterUtil.java:65)
> at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrC
> all.java:806)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
> atchFilter.java:382)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
> atchFilter.java:326)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
> r(ServletHandler.java:1751)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
> dler.java:582)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
> Handler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
> ndler.java:548)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(
> SessionHandler.java:226)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
> ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
> ler.java:512)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(
> SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(
> ContextHandler.java:1112)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
> Handler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
> ndle(ContextHandlerCollection.java:213)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(
> HandlerCollection.java:119)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
> erWrapper.java:134)
> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
> iteHandler.java:335)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
> erWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
> ction.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
> succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
> elEndPoint.java:93)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
> .executeProduceConsume(ExecuteProduceConsume.java:303)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
> .produceConsume(ExecuteProduceConsume.java:148)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
> .run(ExecuteProduceConsume.java:136)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
> ThreadPool.java:671)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
> hreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://vesta:9100/solr/MODEL10
> 24_1522696624083_shard20_replica_n75: Expected mime type
> application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 404 Not Found</title>
> </head>
> <body><h2>HTTP ERROR 404</h2>
> <p>Problem accessing /solr/MODEL1024_1522696624083_shard20_replica_n75/select.
> Reason:
> <pre> Not Found</pre></p>
> </body>
> </html>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.solr.client.solrj.io.stream.TextLogitStream.read(
> TextLogitStream.java:459)
> ... 47 more
> Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://vesta:9100/solr/MODEL10
> 24_1522696624083_shard20_replica_n75: Expected mime type
> application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 404 Not Found</title>
> </head>
> <body><h2>HTTP ERROR 404</h2>
> <p>Problem accessing /solr/MODEL1024_1522696624083_shard20_replica_n75/select.
> Reason:
> <pre> Not Found</pre></p>
> </body>
> </html>
>
> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
> od(HttpSolrClient.java:590)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
> pSolrClient.java:253)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
> pSolrClient.java:242)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest
> .java:178)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest
> .java:195)
> at org.apache.solr.client.solrj.io.stream.TextLogitStream$Logit
> Call.call(TextLogitStream.java:640)
> at org.apache.solr.client.solrj.io.stream.TextLogitStream$Logit
> Call.call(TextLogitStream.java:582)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
> xecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:624)
> ... 1 more
> Last Check: 4/2/2018, 3:47:15 PM
>
> Thank you!
>
> -Joe Obernberger
>
>
Re: Solr 7.1.0 - concurrent.ExecutionException building model
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Joel. I gave each node in the cluster 24g of heap and it ran,
but then failed on the 50th iteration (was trying to do 1,000).
This time, I have the error on the node and the exception from the
client running the stream command. The node (Doris) has 3 errors that
occurred at the same time logged:
java.io.IOException: java.io.IOException:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.read(TextLogitStream.java:498)
at
org.apache.solr.client.solrj.io.stream.PushBackStream.read(PushBackStream.java:87)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.read(UpdateStream.java:109)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:68)
at
org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:627)
at
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:87)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:84)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.lang.IndexOutOfBoundsException:
Index: 0, Size: 0
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.getShardUrls(TextLogitStream.java:365)
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.read(TextLogitStream.java:457)
... 47 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.getShardUrls(TextLogitStream.java:357)
Then:
CloudSolrClient
Request to collection models failed due to (0) java.net.SocketException:
Socket closed,​ retry? 0
Then a warning:
org.apache.solr.client.solrj.SolrServerException:
java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:976)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:922)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.uploadBatchToCollection(UpdateStream.java:299)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.read(UpdateStream.java:124)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:68)
at
org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:627)
at
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:87)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:84)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:173)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:516)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
... 52 more
And finally:
CloudSolrClient
Request to collection models failed due to (0)
java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,​
pool size = 0,​ active threads = 0,​ queued tasks =
0,​ completed tasks = 3],​ retry? 1
java.io.IOException: Unexpected error when adding documents to
collection models-
org.apache.solr.client.solrj.SolrServerException:java.util.concurrent.RejectedExecutionException:
Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at
org.apache.solr.client.solrj.io.stream.UpdateStream.uploadBatchToCollection(UpdateStream.java:304)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.read(UpdateStream.java:124)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:68)
at
org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:627)
at
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:87)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:84)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
The client has:
java.io.IOException: java.util.concurrent.ExecutionException:
java.io.IOException: params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522969708719,features(MODEL1033_1522969708719,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522969708719",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:405)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(CloudSolrStream.java:275)
at
com.ngc.bigdata.ie_solrmodelbuilder.SolrModelBuilderProcessor.doWork(SolrModelBuilderProcessor.java:122)
at
com.ngc.intelenterprise.intelentutil.utils.Processor.run(Processor.java:140)
at
com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.process(IntelEntQueueProc.java:208)
at
org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:114)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:699)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:637)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:605)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:246)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1144)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1136)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1033)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522969708719,features(MODEL1033_1522969708719,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522969708719",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:399)
... 27 more
Caused by: java.io.IOException: params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522969708719,features(MODEL1033_1522969708719,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522969708719",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at
org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:115)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:510)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
... 3 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
occured while waiting response from server at:
http://doris:9100/solr/MODEL1033_1522969708719_shard30_replica_n113
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:637)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at
org.apache.solr.client.solrj.io.stream.SolrStream.constructParser(SolrStream.java:269)
at
org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:113)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:525)
... 12 more
The cluster was basically idle during this run, but that certainly
doesn't mean there was not a legitimate read timeout. Thanks for looking!
-Joe
On 4/5/2018 8:27 PM, Joel Bernstein wrote:
> Hi Joe,
>
> Currently you will eventually run into memory problems if the training sets
> gets too large. Under the covers on each node it is creating a matrix with
> a row for each document and a column for each feature. This can get large
> quite quickly. By choosing fewer features you can make this matrix much
> smaller.
>
> Its fairly easy to make the train function work on a random sample of the
> training set on each iteration rather then the entire training set, but
> currently this is not how its implemented. Feel free to create a ticket
> requesting this the sampling approach.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Apr 5, 2018 at 5:32 PM, Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> I tried to build a large model based on about 1.2 million documents. One
>> of the nodes ran out of memory and killed itself. Is this much data not
>> reasonable to use? The nodes have 16g of heap. Happy to increase it, but
>> not sure if this is possible?
>>
>> Thank you!
>>
>> -Joe
>>
>>
>>
>> On 4/5/2018 10:24 AM, Joe Obernberger wrote:
>>
>>> Thank you Shawn - sorry so long to respond, been playing around with this
>>> a good bit. It is an amazing capability. It looks like it could be
>>> related to certain nodes in the cluster not responding quickly enough. In
>>> one case, I got the concurrent.ExecutionException, but it looks like the
>>> root cause was a SocketTimeoutException. I'm using HDFS for the index and
>>> it gets hit pretty hard by other processes running, and I'm wondering if
>>> that's causing this.
>>>
>>> java.io.IOException: java.util.concurrent.ExecutionException:
>>> java.io.IOException: params expr=update(models,+batchSize%
>>> 3D"50",train(MODEL1033_1522883727011,features(MODEL1033_
>>> 1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_
>>> 1522883727011",field%3D"Text",outcome%3D"out_i",
>>> positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL10
>>> 33",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000")
>>> )&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
>>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openS
>>> treams(CloudSolrStream.java:405)
>>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(
>>> CloudSolrStream.java:275)
>>> at com.ngc.bigdata.ie_solrmodelbuilder.SolrModelBuilderProcesso
>>> r.doWork(SolrModelBuilderProcessor.java:114)
>>> at com.ngc.intelenterprise.intelentutil.utils.Processor.run(
>>> Processor.java:140)
>>> at com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.
>>> process(IntelEntQueueProc.java:208)
>>> at org.apache.camel.processor.DelegateSyncProcessor.process(Del
>>> egateSyncProcessor.java:63)
>>> at org.apache.camel.management.InstrumentationProcessor.process
>>> (InstrumentationProcessor.java:77)
>>> at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
>>> deliveryErrorHandler.java:460)
>>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>>> melInternalProcessor.java:190)
>>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>>> melInternalProcessor.java:190)
>>> at org.apache.camel.component.direct.DirectProducer.process(Dir
>>> ectProducer.java:62)
>>> at org.apache.camel.processor.SendProcessor.process(SendProcess
>>> or.java:141)
>>> at org.apache.camel.management.InstrumentationProcessor.process
>>> (InstrumentationProcessor.java:77)
>>> at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
>>> deliveryErrorHandler.java:460)
>>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>>> melInternalProcessor.java:190)
>>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>>> melInternalProcessor.java:190)
>>> at org.apache.camel.component.jms.EndpointMessageListener.onMes
>>> sage(EndpointMessageListener.java:114)
>>> at org.springframework.jms.listener.AbstractMessageListenerCont
>>> ainer.doInvokeListener(AbstractMessageListenerContainer.java:699)
>>> at org.springframework.jms.listener.AbstractMessageListenerCont
>>> ainer.invokeListener(AbstractMessageListenerContainer.java:637)
>>> at org.springframework.jms.listener.AbstractMessageListenerCont
>>> ainer.doExecuteListener(AbstractMessageListenerContainer.java:605)
>>> at org.springframework.jms.listener.AbstractPollingMessageListe
>>> nerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.
>>> java:308)
>>> at org.springframework.jms.listener.AbstractPollingMessageListe
>>> nerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.
>>> java:246)
>>> at org.springframework.jms.listener.DefaultMessageListenerConta
>>> iner$AsyncMessageListenerInvoker.invokeListener(DefaultMessageLis
>>> tenerContainer.java:1144)
>>> at org.springframework.jms.listener.DefaultMessageListenerConta
>>> iner$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessag
>>> eListenerContainer.java:1136)
>>> at org.springframework.jms.listener.DefaultMessageListenerConta
>>> iner$AsyncMessageListenerInvoker.run(DefaultMessageListenerContai
>>> ner.java:1033)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1149)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
>>> params expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883
>>> 727011,features(MODEL1033_1522883727011,q%3D"*:*",
>>> featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",
>>> outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1
>>> 000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"
>>> out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:
>>> *&fl=id&sort=id+asc&distrib=false
>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openS
>>> treams(CloudSolrStream.java:399)
>>> ... 27 more
>>> Caused by: java.io.IOException: params expr=update(models,+batchSize%
>>> 3D"50",train(MODEL1033_1522883727011,features(MODEL1033_
>>> 1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_
>>> 1522883727011",field%3D"Text",outcome%3D"out_i",
>>> positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL10
>>> 33",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000")
>>> )&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
>>> at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrS
>>> tream.java:115)
>>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream$Strea
>>> mOpener.call(CloudSolrStream.java:510)
>>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream$Strea
>>> mOpener.call(CloudSolrStream.java:499)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
>>> xecutor.lambda$execute$0(ExecutorUtil.java:188)
>>> ... 3 more
>>> Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
>>> occured while waiting response from server at:
>>> http://leda:9100/solr/MODEL1033_1522883727011_shard20_replica_n74
>>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
>>> od(HttpSolrClient.java:637)
>>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>>> pSolrClient.java:253)
>>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>>> pSolrClient.java:242)
>>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.j
>>> ava:1219)
>>> at org.apache.solr.client.solrj.io.stream.SolrStream.constructP
>>> arser(SolrStream.java:269)
>>> at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrS
>>> tream.java:113)
>>> ... 7 more
>>> Caused by: java.net.SocketTimeoutException: Read timed out
>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>> at java.net.SocketInputStream.socketRead(SocketInputStream.java
>>> :116)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:171)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:141)
>>> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(Se
>>> ssionInputBufferImpl.java:139)
>>> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(Se
>>> ssionInputBufferImpl.java:155)
>>> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(Sess
>>> ionInputBufferImpl.java:284)
>>> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
>>> d(DefaultHttpResponseParser.java:138)
>>> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
>>> d(DefaultHttpResponseParser.java:56)
>>> at org.apache.http.impl.io.AbstractMessageParser.parse(Abstract
>>> MessageParser.java:261)
>>> at org.apache.http.impl.DefaultBHttpClientConnection.receiveRes
>>> ponseHeader(DefaultBHttpClientConnection.java:165)
>>> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(C
>>> PoolProxy.java:165)
>>> at org.apache.http.protocol.HttpRequestExecutor.doReceiveRespon
>>> se(HttpRequestExecutor.java:272)
>>> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpReq
>>> uestExecutor.java:124)
>>> at org.apache.http.impl.execchain.MainClientExec.execute(
>>> MainClientExec.java:272)
>>> at org.apache.http.impl.execchain.ProtocolExec.execute(
>>> ProtocolExec.java:185)
>>> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.
>>> java:89)
>>> at org.apache.http.impl.execchain.RedirectExec.execute(
>>> RedirectExec.java:111)
>>> at org.apache.http.impl.client.InternalHttpClient.doExecute(Int
>>> ernalHttpClient.java:185)
>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>> eableHttpClient.java:83)
>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>> eableHttpClient.java:56)
>>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
>>> od(HttpSolrClient.java:525)
>>>
>>> -Joe
>>>
>>>
>>> On 4/2/2018 7:09 PM, Shawn Heisey wrote:
>>>
>>>> On 4/2/2018 1:55 PM, Joe Obernberger wrote:
>>>>
>>>>> The training data was split across 20 shards - specifically created
>>>>> with:
>>>>> http://icarus.querymasters.com:9100/solr/admin/collections?
>>>>> action=CREATE&name=MODEL1024_1522696624083&numShards=20&rep
>>>>> licationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
>>>>>
>>>>> Any ideas? The complete error is:
>>>>>
>>>> <snip>
>>>>
>>>>> <body><h2>HTTP ERROR 404</h2>
>>>>> <p>Problem accessing
>>>>> /solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
>>>>> <pre> Not Found</pre></p>
>>>>> </body>
>>>>>
>>>> I'll warn you in advance that I know nothing at all about the learning
>>>> to rank functionality. I'm replying about the underlying error you're
>>>> getting, independent of what your query is trying to accomplish.
>>>>
>>>> It's a 404 error, trying to access the URL mentioned above.
>>>>
>>>> The error doesn't indicate exactly WHAT wasn't found. It could either
>>>> be the core named "MODEL1024_1522696624083_shard20_replica_n75" or the
>>>> "/select" handler on that core. That's something you need to figure
>>>> out. It could be that the core *does* exist, but for some reason, Solr
>>>> on that machine was unable to start it.
>>>>
>>>> The solr.log file on the Solr instance that returned the error (which
>>>> seems to be on the machine named vesta, answering to port 9100) may have
>>>> more detail for the error, or some additional error messages.
>>>>
>>>> Normally SolrCloud is good at making sure that requests aren't sent to
>>>> resources that aren't working. So I'm not sure why this happened.
>>>>
>>>> Are there other errors or warnings in the solr.log file, either on the
>>>> instance where you sent your request, or the instance that returned the
>>>> 404 error?
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>>>
>>>> ---
>>>> This email has been checked for viruses by AVG.
>>>> http://www.avg.com
>>>>
>>>>
Re: Solr 7.1.0 - concurrent.ExecutionException building model
Posted by Joel Bernstein <jo...@gmail.com>.
Hi Joe,
Currently you will eventually run into memory problems if the training sets
gets too large. Under the covers on each node it is creating a matrix with
a row for each document and a column for each feature. This can get large
quite quickly. By choosing fewer features you can make this matrix much
smaller.
Its fairly easy to make the train function work on a random sample of the
training set on each iteration rather then the entire training set, but
currently this is not how its implemented. Feel free to create a ticket
requesting this the sampling approach.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Apr 5, 2018 at 5:32 PM, Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> I tried to build a large model based on about 1.2 million documents. One
> of the nodes ran out of memory and killed itself. Is this much data not
> reasonable to use? The nodes have 16g of heap. Happy to increase it, but
> not sure if this is possible?
>
> Thank you!
>
> -Joe
>
>
>
> On 4/5/2018 10:24 AM, Joe Obernberger wrote:
>
>> Thank you Shawn - sorry so long to respond, been playing around with this
>> a good bit. It is an amazing capability. It looks like it could be
>> related to certain nodes in the cluster not responding quickly enough. In
>> one case, I got the concurrent.ExecutionException, but it looks like the
>> root cause was a SocketTimeoutException. I'm using HDFS for the index and
>> it gets hit pretty hard by other processes running, and I'm wondering if
>> that's causing this.
>>
>> java.io.IOException: java.util.concurrent.ExecutionException:
>> java.io.IOException: params expr=update(models,+batchSize%
>> 3D"50",train(MODEL1033_1522883727011,features(MODEL1033_
>> 1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_
>> 1522883727011",field%3D"Text",outcome%3D"out_i",
>> positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL10
>> 33",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000")
>> )&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openS
>> treams(CloudSolrStream.java:405)
>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(
>> CloudSolrStream.java:275)
>> at com.ngc.bigdata.ie_solrmodelbuilder.SolrModelBuilderProcesso
>> r.doWork(SolrModelBuilderProcessor.java:114)
>> at com.ngc.intelenterprise.intelentutil.utils.Processor.run(
>> Processor.java:140)
>> at com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.
>> process(IntelEntQueueProc.java:208)
>> at org.apache.camel.processor.DelegateSyncProcessor.process(Del
>> egateSyncProcessor.java:63)
>> at org.apache.camel.management.InstrumentationProcessor.process
>> (InstrumentationProcessor.java:77)
>> at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
>> deliveryErrorHandler.java:460)
>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>> melInternalProcessor.java:190)
>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>> melInternalProcessor.java:190)
>> at org.apache.camel.component.direct.DirectProducer.process(Dir
>> ectProducer.java:62)
>> at org.apache.camel.processor.SendProcessor.process(SendProcess
>> or.java:141)
>> at org.apache.camel.management.InstrumentationProcessor.process
>> (InstrumentationProcessor.java:77)
>> at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
>> deliveryErrorHandler.java:460)
>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>> melInternalProcessor.java:190)
>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>> melInternalProcessor.java:190)
>> at org.apache.camel.component.jms.EndpointMessageListener.onMes
>> sage(EndpointMessageListener.java:114)
>> at org.springframework.jms.listener.AbstractMessageListenerCont
>> ainer.doInvokeListener(AbstractMessageListenerContainer.java:699)
>> at org.springframework.jms.listener.AbstractMessageListenerCont
>> ainer.invokeListener(AbstractMessageListenerContainer.java:637)
>> at org.springframework.jms.listener.AbstractMessageListenerCont
>> ainer.doExecuteListener(AbstractMessageListenerContainer.java:605)
>> at org.springframework.jms.listener.AbstractPollingMessageListe
>> nerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.
>> java:308)
>> at org.springframework.jms.listener.AbstractPollingMessageListe
>> nerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.
>> java:246)
>> at org.springframework.jms.listener.DefaultMessageListenerConta
>> iner$AsyncMessageListenerInvoker.invokeListener(DefaultMessageLis
>> tenerContainer.java:1144)
>> at org.springframework.jms.listener.DefaultMessageListenerConta
>> iner$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessag
>> eListenerContainer.java:1136)
>> at org.springframework.jms.listener.DefaultMessageListenerConta
>> iner$AsyncMessageListenerInvoker.run(DefaultMessageListenerContai
>> ner.java:1033)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
>> params expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883
>> 727011,features(MODEL1033_1522883727011,q%3D"*:*",
>> featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",
>> outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1
>> 000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"
>> out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:
>> *&fl=id&sort=id+asc&distrib=false
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openS
>> treams(CloudSolrStream.java:399)
>> ... 27 more
>> Caused by: java.io.IOException: params expr=update(models,+batchSize%
>> 3D"50",train(MODEL1033_1522883727011,features(MODEL1033_
>> 1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_
>> 1522883727011",field%3D"Text",outcome%3D"out_i",
>> positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL10
>> 33",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000")
>> )&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
>> at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrS
>> tream.java:115)
>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream$Strea
>> mOpener.call(CloudSolrStream.java:510)
>> at org.apache.solr.client.solrj.io.stream.CloudSolrStream$Strea
>> mOpener.call(CloudSolrStream.java:499)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
>> xecutor.lambda$execute$0(ExecutorUtil.java:188)
>> ... 3 more
>> Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
>> occured while waiting response from server at:
>> http://leda:9100/solr/MODEL1033_1522883727011_shard20_replica_n74
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
>> od(HttpSolrClient.java:637)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>> pSolrClient.java:253)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>> pSolrClient.java:242)
>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.j
>> ava:1219)
>> at org.apache.solr.client.solrj.io.stream.SolrStream.constructP
>> arser(SolrStream.java:269)
>> at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrS
>> tream.java:113)
>> ... 7 more
>> Caused by: java.net.SocketTimeoutException: Read timed out
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.socketRead(SocketInputStream.java
>> :116)
>> at java.net.SocketInputStream.read(SocketInputStream.java:171)
>> at java.net.SocketInputStream.read(SocketInputStream.java:141)
>> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(Se
>> ssionInputBufferImpl.java:139)
>> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(Se
>> ssionInputBufferImpl.java:155)
>> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(Sess
>> ionInputBufferImpl.java:284)
>> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
>> d(DefaultHttpResponseParser.java:138)
>> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
>> d(DefaultHttpResponseParser.java:56)
>> at org.apache.http.impl.io.AbstractMessageParser.parse(Abstract
>> MessageParser.java:261)
>> at org.apache.http.impl.DefaultBHttpClientConnection.receiveRes
>> ponseHeader(DefaultBHttpClientConnection.java:165)
>> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(C
>> PoolProxy.java:165)
>> at org.apache.http.protocol.HttpRequestExecutor.doReceiveRespon
>> se(HttpRequestExecutor.java:272)
>> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpReq
>> uestExecutor.java:124)
>> at org.apache.http.impl.execchain.MainClientExec.execute(
>> MainClientExec.java:272)
>> at org.apache.http.impl.execchain.ProtocolExec.execute(
>> ProtocolExec.java:185)
>> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.
>> java:89)
>> at org.apache.http.impl.execchain.RedirectExec.execute(
>> RedirectExec.java:111)
>> at org.apache.http.impl.client.InternalHttpClient.doExecute(Int
>> ernalHttpClient.java:185)
>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>> eableHttpClient.java:83)
>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>> eableHttpClient.java:56)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
>> od(HttpSolrClient.java:525)
>>
>> -Joe
>>
>>
>> On 4/2/2018 7:09 PM, Shawn Heisey wrote:
>>
>>> On 4/2/2018 1:55 PM, Joe Obernberger wrote:
>>>
>>>> The training data was split across 20 shards - specifically created
>>>> with:
>>>> http://icarus.querymasters.com:9100/solr/admin/collections?
>>>> action=CREATE&name=MODEL1024_1522696624083&numShards=20&rep
>>>> licationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
>>>>
>>>> Any ideas? The complete error is:
>>>>
>>> <snip>
>>>
>>>> <body><h2>HTTP ERROR 404</h2>
>>>> <p>Problem accessing
>>>> /solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
>>>> <pre> Not Found</pre></p>
>>>> </body>
>>>>
>>> I'll warn you in advance that I know nothing at all about the learning
>>> to rank functionality. I'm replying about the underlying error you're
>>> getting, independent of what your query is trying to accomplish.
>>>
>>> It's a 404 error, trying to access the URL mentioned above.
>>>
>>> The error doesn't indicate exactly WHAT wasn't found. It could either
>>> be the core named "MODEL1024_1522696624083_shard20_replica_n75" or the
>>> "/select" handler on that core. That's something you need to figure
>>> out. It could be that the core *does* exist, but for some reason, Solr
>>> on that machine was unable to start it.
>>>
>>> The solr.log file on the Solr instance that returned the error (which
>>> seems to be on the machine named vesta, answering to port 9100) may have
>>> more detail for the error, or some additional error messages.
>>>
>>> Normally SolrCloud is good at making sure that requests aren't sent to
>>> resources that aren't working. So I'm not sure why this happened.
>>>
>>> Are there other errors or warnings in the solr.log file, either on the
>>> instance where you sent your request, or the instance that returned the
>>> 404 error?
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>> ---
>>> This email has been checked for viruses by AVG.
>>> http://www.avg.com
>>>
>>>
>>
>
Re: Solr 7.1.0 - concurrent.ExecutionException building model
Posted by Joe Obernberger <jo...@gmail.com>.
I tried to build a large model based on about 1.2 million documents.
One of the nodes ran out of memory and killed itself. Is this much data
not reasonable to use? The nodes have 16g of heap. Happy to increase
it, but not sure if this is possible?
Thank you!
-Joe
On 4/5/2018 10:24 AM, Joe Obernberger wrote:
> Thank you Shawn - sorry so long to respond, been playing around with
> this a good bit. It is an amazing capability. It looks like it could
> be related to certain nodes in the cluster not responding quickly
> enough. In one case, I got the concurrent.ExecutionException, but it
> looks like the root cause was a SocketTimeoutException. I'm using
> HDFS for the index and it gets hit pretty hard by other processes
> running, and I'm wondering if that's causing this.
>
> java.io.IOException: java.util.concurrent.ExecutionException:
> java.io.IOException: params
> expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883727011,features(MODEL1033_1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
> at
> org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:405)
> at
> org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(CloudSolrStream.java:275)
> at
> com.ngc.bigdata.ie_solrmodelbuilder.SolrModelBuilderProcessor.doWork(SolrModelBuilderProcessor.java:114)
> at
> com.ngc.intelenterprise.intelentutil.utils.Processor.run(Processor.java:140)
> at
> com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.process(IntelEntQueueProc.java:208)
> at
> org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at
> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
> at
> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at
> org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:114)
> at
> org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:699)
> at
> org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:637)
> at
> org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:605)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:246)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1144)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1136)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1033)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.io.IOException: params
> expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883727011,features(MODEL1033_1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:399)
> ... 27 more
> Caused by: java.io.IOException: params
> expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883727011,features(MODEL1033_1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
> at
> org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:115)
> at
> org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:510)
> at
> org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:499)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> ... 3 more
> Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
> occured while waiting response from server at:
> http://leda:9100/solr/MODEL1033_1522883727011_shard20_replica_n74
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:637)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
> at
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
> at
> org.apache.solr.client.solrj.io.stream.SolrStream.constructParser(SolrStream.java:269)
> at
> org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:113)
> ... 7 more
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:525)
>
> -Joe
>
>
> On 4/2/2018 7:09 PM, Shawn Heisey wrote:
>> On 4/2/2018 1:55 PM, Joe Obernberger wrote:
>>> The training data was split across 20 shards - specifically created
>>> with:
>>> http://icarus.querymasters.com:9100/solr/admin/collections?action=CREATE&name=MODEL1024_1522696624083&numShards=20&replicationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
>>>
>>>
>>> Any ideas? The complete error is:
>> <snip>
>>> <body><h2>HTTP ERROR 404</h2>
>>> <p>Problem accessing
>>> /solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
>>> <pre> Not Found</pre></p>
>>> </body>
>> I'll warn you in advance that I know nothing at all about the learning
>> to rank functionality. I'm replying about the underlying error you're
>> getting, independent of what your query is trying to accomplish.
>>
>> It's a 404 error, trying to access the URL mentioned above.
>>
>> The error doesn't indicate exactly WHAT wasn't found. It could either
>> be the core named "MODEL1024_1522696624083_shard20_replica_n75" or the
>> "/select" handler on that core. That's something you need to figure
>> out. It could be that the core *does* exist, but for some reason, Solr
>> on that machine was unable to start it.
>>
>> The solr.log file on the Solr instance that returned the error (which
>> seems to be on the machine named vesta, answering to port 9100) may have
>> more detail for the error, or some additional error messages.
>>
>> Normally SolrCloud is good at making sure that requests aren't sent to
>> resources that aren't working. So I'm not sure why this happened.
>>
>> Are there other errors or warnings in the solr.log file, either on the
>> instance where you sent your request, or the instance that returned the
>> 404 error?
>>
>> Thanks,
>> Shawn
>>
>>
>> ---
>> This email has been checked for viruses by AVG.
>> http://www.avg.com
>>
>
Re: Solr 7.1.0 - concurrent.ExecutionException building model
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Shawn - sorry so long to respond, been playing around with
this a good bit. It is an amazing capability. It looks like it could
be related to certain nodes in the cluster not responding quickly
enough. In one case, I got the concurrent.ExecutionException, but it
looks like the root cause was a SocketTimeoutException. I'm using HDFS
for the index and it gets hit pretty hard by other processes running,
and I'm wondering if that's causing this.
java.io.IOException: java.util.concurrent.ExecutionException:
java.io.IOException: params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883727011,features(MODEL1033_1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:405)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(CloudSolrStream.java:275)
at
com.ngc.bigdata.ie_solrmodelbuilder.SolrModelBuilderProcessor.doWork(SolrModelBuilderProcessor.java:114)
at
com.ngc.intelenterprise.intelentutil.utils.Processor.run(Processor.java:140)
at
com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.process(IntelEntQueueProc.java:208)
at
org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:114)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:699)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:637)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:605)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:246)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1144)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1136)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1033)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883727011,features(MODEL1033_1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:399)
... 27 more
Caused by: java.io.IOException: params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883727011,features(MODEL1033_1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at
org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:115)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:510)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
... 3 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
occured while waiting response from server at:
http://leda:9100/solr/MODEL1033_1522883727011_shard20_replica_n74
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:637)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at
org.apache.solr.client.solrj.io.stream.SolrStream.constructParser(SolrStream.java:269)
at
org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:113)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:525)
-Joe
On 4/2/2018 7:09 PM, Shawn Heisey wrote:
> On 4/2/2018 1:55 PM, Joe Obernberger wrote:
>> The training data was split across 20 shards - specifically created with:
>> http://icarus.querymasters.com:9100/solr/admin/collections?action=CREATE&name=MODEL1024_1522696624083&numShards=20&replicationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
>>
>> Any ideas? The complete error is:
> <snip>
>> <body><h2>HTTP ERROR 404</h2>
>> <p>Problem accessing
>> /solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
>> <pre> Not Found</pre></p>
>> </body>
> I'll warn you in advance that I know nothing at all about the learning
> to rank functionality. I'm replying about the underlying error you're
> getting, independent of what your query is trying to accomplish.
>
> It's a 404 error, trying to access the URL mentioned above.
>
> The error doesn't indicate exactly WHAT wasn't found. It could either
> be the core named "MODEL1024_1522696624083_shard20_replica_n75" or the
> "/select" handler on that core. That's something you need to figure
> out. It could be that the core *does* exist, but for some reason, Solr
> on that machine was unable to start it.
>
> The solr.log file on the Solr instance that returned the error (which
> seems to be on the machine named vesta, answering to port 9100) may have
> more detail for the error, or some additional error messages.
>
> Normally SolrCloud is good at making sure that requests aren't sent to
> resources that aren't working. So I'm not sure why this happened.
>
> Are there other errors or warnings in the solr.log file, either on the
> instance where you sent your request, or the instance that returned the
> 404 error?
>
> Thanks,
> Shawn
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>
Re: Solr 7.1.0 - concurrent.ExecutionException building model
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/2/2018 1:55 PM, Joe Obernberger wrote:
> The training data was split across 20 shards - specifically created with:
> http://icarus.querymasters.com:9100/solr/admin/collections?action=CREATE&name=MODEL1024_1522696624083&numShards=20&replicationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
>
> Any ideas? The complete error is:
<snip>
> <body><h2>HTTP ERROR 404</h2>
> <p>Problem accessing
> /solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
> <pre> Not Found</pre></p>
> </body>
I'll warn you in advance that I know nothing at all about the learning
to rank functionality. I'm replying about the underlying error you're
getting, independent of what your query is trying to accomplish.
It's a 404 error, trying to access the URL mentioned above.
The error doesn't indicate exactly WHAT wasn't found. It could either
be the core named "MODEL1024_1522696624083_shard20_replica_n75" or the
"/select" handler on that core. That's something you need to figure
out. It could be that the core *does* exist, but for some reason, Solr
on that machine was unable to start it.
The solr.log file on the Solr instance that returned the error (which
seems to be on the machine named vesta, answering to port 9100) may have
more detail for the error, or some additional error messages.
Normally SolrCloud is good at making sure that requests aren't sent to
resources that aren't working. So I'm not sure why this happened.
Are there other errors or warnings in the solr.log file, either on the
instance where you sent your request, or the instance that returned the
404 error?
Thanks,
Shawn