You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by harjagsbby <ha...@bestbuy.com> on 2019/07/16 20:07:18 UTC
Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core
with Exceptions
2019-07-09 17:37:30.053 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore [promotion_rules-v1_shard1_replica_n1]
**CLOSING SolrCore org.apache.solr.core.SolrCore**@44de0f89
2019-07-09 17:37:30.054 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.core.promotion_rules-v1.shard1.replica_n1, tag=44de0f89
2019-07-09 17:37:30.056 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter
[org.apache.solr.metrics.reporters.SolrJmxReporter@36a60356: rootName =
null, domain = solr.core.promotion_rules-v1.shard1.replica_n1, service url =
null, agent id = null] for registry
solr.core.promotion_rules-v1.shard1.replica_n1 /
com.codahale.metrics.MetricRegistry@79be463d
2019-07-09 17:37:30.148 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.collection.promotion_rules-v1.shard1.leader, tag=44de0f89
2019-07-09 17:37:30.159 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.u.p.DocExpirationUpdateProcessorFactory
Triggering Graceful close of DocExpiration Executor
2019-07-09 17:38:03.062 **ERROR
(facetExecutor-67-thread-26-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-4] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org**
2019-07-09 17:38:03.067 ERROR
(facetExecutor-67-thread-28-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] **o.a.s.c.SolrCore Too many close [count:-5] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org**
2019-07-09 17:38:06.176 ERROR
(facetExecutor-67-thread-25-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-7] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:38:07.602 ERROR
(facetExecutor-67-thread-24-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-6] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:38:11.868 ERROR
(facetExecutor-67-thread-23-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-8] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:38:32.522 ERROR
(facetExecutor-67-thread-4-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] **o.a.s.c.SolrCore Too many close [count:-9] on**
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:39:08.862 ERROR
(facetExecutor-67-thread-3-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-10] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:39:08.988 ERROR
(facetExecutor-67-thread-6-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-12] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:39:09.048 ERROR
(facetExecutor-67-thread-5-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-13] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:39:10.474 ERROR
(facetExecutor-67-thread-2-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-11] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:39:10.572 ERROR
(facetExecutor-67-thread-7-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-14] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
2019-07-09 17:39:10.646 ERROR
(facetExecutor-67-thread-9-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-15] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
solr-user@lucene.apache.org
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr
core with Exceptions
Posted by harjagsbby <ha...@bestbuy.com>.
Few deatils added.
public void close() {
int count = refCount.decrementAndGet();
if (count > 0) return; // close is called often, and only actually
closes if nothing is using it.
if (count < 0) {
log.error("Too many close [count:{}] on {}. Please report this
exception to solr-user@lucene.apache.org", count, this );
assert false : "Too many closes on SolrCore";
return;
}
log.info("{} CLOSING SolrCore {}", logid, this);
// stop reporting metrics
try {
coreMetricManager.close();
} catch (Throwable e) {
SolrException.log(log, e);
if (e instanceof Error) {
throw (Error) e;
}
}}
The above code has refCount in SolrCore.java this atomicRef is in negative
when we do multi threaded facets and joins.
Can anyone explain the semaphore implemenattion here. What would be scenario
when this can go below 0. I can clearly see each thread of multithreaded
facet trying to close the core.
Every Thread of facet tries to call createWeight of JoinQParserPlugin and
attaches closeHooks which getscalled from SolrRequestInfo.clearRequestInfo()
.
if (fromRef != null) {
final RefCounted<SolrIndexSearcher> ref = fromRef;
info.addCloseHook(new Closeable() {
@Override
public void close() {
ref.decref();
}
});
}
info.addCloseHook(new Closeable() {
@Override
public void close() {
fromCore.close();
}
});
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core
with Exceptions
Posted by Mikhail Khludnev <mk...@apache.org>.
Raised https://issues.apache.org/jira/browse/SOLR-13715
On Fri, Aug 23, 2019 at 12:20 PM harjagsbby <ha...@bestbuy.com>
wrote:
> I dint quite get your question.Our Solr collection under contention is not
> shareded. Data being joint is collocated.INfact multithreaded facet help
> our
> performance.
>
> Afer debugging more solr 7 calls JoinQParserPlugin createWeight is called
> by
> each thread of multithreaded facets but not in solr 6. Is this something to
> do with changes to ExecutorUtils awaitTermination?
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
--
Sincerely yours
Mikhail Khludnev
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr
core with Exceptions
Posted by harjagsbby <ha...@bestbuy.com>.
I dint quite get your question.Our Solr collection under contention is not
shareded. Data being joint is collocated.INfact multithreaded facet help our
performance.
Afer debugging more solr 7 calls JoinQParserPlugin createWeight is called by
each thread of multithreaded facets but not in solr 6. Is this something to
do with changes to ExecutorUtils awaitTermination?
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core
with Exceptions
Posted by Mikhail Khludnev <mk...@apache.org>.
This case is worth to be covered with test. Beside of that, if join is
executed in multiple threads it might make bigger footprint that gain from
employing threads. Does it make and impact for consequent searches?
On Thu, Aug 22, 2019 at 8:42 AM harjagsbby <ha...@bestbuy.com>
wrote:
> Few deatils added.
>
> public void close() {
> int count = refCount.decrementAndGet();
> if (count > 0) return; // close is called often, and only actually
> closes if nothing is using it.
> if (count < 0) {
> log.error("Too many close [count:{}] on {}. Please report this
> exception to solr-user@lucene.apache.org", count, this );
> assert false : "Too many closes on SolrCore";
> return;
> }
> log.info("{} CLOSING SolrCore {}", logid, this);
>
> // stop reporting metrics
> try {
> coreMetricManager.close();
> } catch (Throwable e) {
> SolrException.log(log, e);
> if (e instanceof Error) {
> throw (Error) e;
> }
> }}
>
> The above code has refCount in SolrCore.java this atomicRef is in negative
> when we do multi threaded facets and joins.
>
> Can anyone explain the semaphore implemenattion here. What would be
> scenario
> when this can go below 0. I can clearly see each thread of multithreaded
> facet trying to close the core.
>
> Every Thread of facet tries to call createWeight of JoinQParserPlugin and
> attaches closeHooks which getscalled from
> SolrRequestInfo.clearRequestInfo()
> .
>
> if (fromRef != null) {
> final RefCounted<SolrIndexSearcher> ref = fromRef;
> info.addCloseHook(new Closeable() {
> @Override
> public void close() {
> ref.decref();
> }
> });
> }
>
> info.addCloseHook(new Closeable() {
> @Override
> public void close() {
> fromCore.close();
> }
> });
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
--
Sincerely yours
Mikhail Khludnev
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr
core with Exceptions
Posted by harjagsbby <ha...@bestbuy.com>.
Few deatils added.
public void close() {
int count = refCount.decrementAndGet();
if (count > 0) return; // close is called often, and only actually
closes if nothing is using it.
if (count < 0) {
log.error("Too many close [count:{}] on {}. Please report this
exception to solr-user@lucene.apache.org", count, this );
assert false : "Too many closes on SolrCore";
return;
}
log.info("{} CLOSING SolrCore {}", logid, this);
// stop reporting metrics
try {
coreMetricManager.close();
} catch (Throwable e) {
SolrException.log(log, e);
if (e instanceof Error) {
throw (Error) e;
}
}}
The above code has refCount in SolrCore.java this atomicRef is in negative
when we do multi threaded facets and joins.
Can anyone explain the semaphore implemenattion here. What would be scenario
when this can go below 0. I can clearly see each thread of multithreaded
facet trying to close the core.
Every Thread of facet tries to call createWeight of JoinQParserPlugin and
attaches closeHooks which getscalled from SolrRequestInfo.clearRequestInfo()
.
if (fromRef != null) {
final RefCounted<SolrIndexSearcher> ref = fromRef;
info.addCloseHook(new Closeable() {
@Override
public void close() {
ref.decref();
}
});
}
info.addCloseHook(new Closeable() {
@Override
public void close() {
fromCore.close();
}
});
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr
core with Exceptions
Posted by harjagsbby <ha...@bestbuy.com>.
SolrQuery used:
q=*:*&defType=edismax&fl=comboid,skuid&start=0&rows=15&facet.limit=100&variant=A&debugQuery=true&debug.explain.structured=true&fq=!status_facet:inactive&fq=!mode:AF_ONLY&fq=**{!join+from=skuid+to=skuid+fromIndex=promotion_rules}id:promo-opera-test-1000_promo-opera-test-1000**&enableManagedResourceRange=true&facet=true&facet.mincount=1&**facet.threads=32**&facet.field=dynamiccategory_facet&facet.recommend.maxfacets=30&facet.recommend.debug=true&sort=ratingandreviewsort+desc&qt=/promo&timeAllowed=5000
We are migrating to 7.6. We have been using joins with multithreaded facets
for years now.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html