You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "ofer fort (JIRA)" <ji...@apache.org> on 2010/07/08 14:24:49 UTC
[jira] Created: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer when deleing by id
blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id
------------------------------------------------------------------------------
Key: SOLR-1990
URL: https://issues.apache.org/jira/browse/SOLR-1990
Project: Solr
Issue Type: Bug
Components: clients - java
Affects Versions: 1.4.1
Reporter: ofer fort
in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
// this happens for commit...
if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer when deleing by id
Posted by "ofer fort (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886398#action_12886398 ]
ofer fort commented on SOLR-1990:
---------------------------------
reading the thread dump again, it seems i have only one thread deleting and
a lot of adding ones.
the one deleting is just waiting...
> blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id
> ------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer more often then it should
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated SOLR-1990:
---------------------------
Summary: blockUntilFinished() is called in StreamingUpdateSolrServer more often then it should (was: blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id)
Description:
in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document...
{code}
// this happens for commit...
if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
blockUntilFinished();
{code}
...but there are other situations where an UpdateRequest will nave no documents (delete, updates using stream.url or stream.file, etc...)
was:
in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
// this happens for commit...
if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
blockUntilFinished();
updating summary/description to note that the issue goes beyond just deletes by query.
another example noted on the user list is when doing index updates using stream.url or stream.file -- these are update requests that do not include any documents, and even if it is considered intentional that "deletes" block until the queue is empty, these updates certainly shouldn't.
> blockUntilFinished() is called in StreamingUpdateSolrServer more often then it should
> -------------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document...
> {code}
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
> {code}
> ...but there are other situations where an UpdateRequest will nave no documents (delete, updates using stream.url or stream.file, etc...)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer when deleing by id
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886404#action_12886404 ]
Yonik Seeley commented on SOLR-1990:
------------------------------------
Is this behavior by design?
If you do an add followed by a delete of the same document, one would normally expect that the document would be deleted. Without ordering things, you can't really guarantee that though.
Of course the same problem exists if you add multiple documents with the same id - you don't know which will end up winning by being last.
> blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id
> ------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer more often then it should
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899739#action_12899739 ]
Ryan McKinley commented on SOLR-1990:
-------------------------------------
IIRC, the big picture intention is that <commit/> with the StreamingUpdateSolrServer should behave the same as <commit/> with the other (non-streaming) implementations. People using the implementation should not need to know to blockUntilFinished() for commit to work properly.
It seem the check should be:
{code:java}
if( req.getAction() != null ) {
blockUntilFinished();
}
{code}
Though it may be good to try and avoid the two map lookups for every request:
{code:java}
public ACTION getAction() {
if (params==null) return null;
if (params.getBool(UpdateParams.COMMIT, false)) return ACTION.COMMIT;
if (params.getBool(UpdateParams.OPTIMIZE, false)) return ACTION.OPTIMIZE;
return null;
}
{code}
> blockUntilFinished() is called in StreamingUpdateSolrServer more often then it should
> -------------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document...
> {code}
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
> {code}
> ...but there are other situations where an UpdateRequest will nave no documents (delete, updates using stream.url or stream.file, etc...)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer more often then it should
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899130#action_12899130 ]
Hoss Man commented on SOLR-1990:
--------------------------------
Note: the entire premise of the code in question, that updates w/o documents should "blockUntilFinished" because "this happens for commit" makes no sense to me.
In the Runnable where requests that do make it into the queue are processed, there is code that explicitly checks for requests which include the "commit" or "optimize" params and those are processed as part of the stream -- w/o blocking.
So why should a "commit" by itself block, but an update doc that includes an "optimize" not block?
> blockUntilFinished() is called in StreamingUpdateSolrServer more often then it should
> -------------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document...
> {code}
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
> {code}
> ...but there are other situations where an UpdateRequest will nave no documents (delete, updates using stream.url or stream.file, etc...)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer when deleing by id
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886409#action_12886409 ]
Mark Miller commented on SOLR-1990:
-----------------------------------
bq. reading the thread dump again, it seems i have only one thread deleting and a lot of adding ones. the one deleting is just waiting...
Right - the delete thread will wait until all the other threads are done adding.
bq. Is this behavior by design?
I doubt it - looks like an oversight - don't see why the delete should not just go in the queue like doc adds.
> blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id
> ------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer when deleing by id
Posted by "ofer fort (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886389#action_12886389 ]
ofer fort commented on SOLR-1990:
---------------------------------
I am not sure, still investigating.
At a specific moment i had two threads trying to delete two documents, and a
few others that were adding documents.
The ones deleting were both parking(waiting on the lock), so my big concern
is that it's actually a deadlock, but still not sure
two thread with:
"IndexWorker0" prio=5 tid=267 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.blockUntilFinished(StreamingUpdateSolrServer.java:232)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:181)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at
org.apache.solr.client.solrj.SolrServer.deleteById(SolrServer.java:102)
a few threads with:
"pool-1-thread-158" prio=5 tid=1288 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:137)
> blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id
> ------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer when deleing by id
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886371#action_12886371 ]
Mark Miller commented on SOLR-1990:
-----------------------------------
Nice catch - is this causing you a particular problem?
Looks like if you submit a delete in a long line of adds, rather than putting the delete into the request queue, it's going to block and wait for all the previous doc adds to finish, then process the delete, then continue processing docs.
Could always check for a delete query or delete id(s), but I wonder if there is a better way to just check if the request is a commit or optimize rather than checking everything else.
> blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id
> ------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-1990) blockUntilFinished() is called in
StreamingUpdateSolrServer when deleing by id
Posted by "ofer fort (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886422#action_12886422 ]
ofer fort commented on SOLR-1990:
---------------------------------
Each document has it's own id.
The deleted document is not one of the added ones. It was added a while ago
and if my application realizes it is not relevant anymore, it will delete
it.
ב-08/07/2010, בשעה 21:47, "Yonik Seeley (JIRA)" <ji...@apache.org> כתב/ה:
[
https://issues.apache.org/jira/browse/SOLR-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886404#action_12886404]
Yonik Seeley commented on SOLR-1990:
------------------------------------
Is this behavior by design?
If you do an add followed by a delete of the same document, one would
normally expect that the document would be deleted. Without ordering
things, you can't really guarantee that though.
Of course the same problem exists if you add multiple documents with the
same id - you don't know which will end up winning by being last.
blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by
id
------------------------------------------------------------------------------
Key: SOLR-1990
URL: https://issues.apache.org/jira/browse/SOLR-1990
Project: Solr
Issue Type: Bug
Components: clients - java
Affects Versions: 1.4.1
Reporter: ofer fort
Original Estimate: 24h
Remaining Estimate: 24h
in the StreamingUpdateSolrServer .request() it identifies a commit/optimize
request by having no document, but also the delete doesn't have a docuemnt.
// this happens for commit...
if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
> blockUntilFinished() is called in StreamingUpdateSolrServer when deleing by id
> ------------------------------------------------------------------------------
>
> Key: SOLR-1990
> URL: https://issues.apache.org/jira/browse/SOLR-1990
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 1.4.1
> Reporter: ofer fort
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> in the StreamingUpdateSolrServer .request() it identifies a commit/optimize request by having no document, but also the delete doesn't have a docuemnt.
> // this happens for commit...
> if( req.getDocuments()==null || req.getDocuments().isEmpty() ) {
> blockUntilFinished();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org