You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2011/02/01 22:58:28 UTC
[jira] Created: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Lock starvation can cause commit to never run when many clients are adding docs
-------------------------------------------------------------------------------
Key: SOLR-2342
URL: https://issues.apache.org/jira/browse/SOLR-2342
Project: Solr
Issue Type: Bug
Components: update
Reporter: Michael McCandless
Priority: Minor
I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved SOLR-2342.
--------------------------------------
Resolution: Fixed
Fix Version/s: 4.0
Assignee: Michael McCandless
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1, 4.0
>
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated SOLR-2342:
-------------------------------------
Fix Version/s: 3.1
We may as well do this for 3.1?
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994544#comment-12994544 ]
Michael McCandless commented on SOLR-2342:
------------------------------------------
Thanks Mark; I'll commit shortly...
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993770#comment-12993770 ]
Michael McCandless commented on SOLR-2342:
------------------------------------------
OK, thanks Yonik. That's good news. Should we commit the fairness?
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991965#comment-12991965 ]
Michael McCandless commented on SOLR-2342:
------------------------------------------
OK, passing true to the ReentrantReadWriteLock fixes the starvation in this test... Should we commit that?
Passing true does make the read lock acq more costly, but I suspect this is in the noise for typical indexing.
And, while I think the stress test is rather unnatural (normally one thread should call commit in the end), I fear many apps may (for simplicity) do something similar to this stress test. Also, the fact that auto-commit is also starved is I think a more realistic failure...
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992245#comment-12992245 ]
Yonik Seeley commented on SOLR-2342:
------------------------------------
bq. Passing true does make the read lock acq more costly, but I suspect this is in the noise for typical indexing.
I'll try to do a quick test to verify this soon.
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994173#comment-12994173 ]
Mark Miller commented on SOLR-2342:
-----------------------------------
bq. We may as well do this for 3.1?
+1
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993331#comment-12993331 ]
Yonik Seeley commented on SOLR-2342:
------------------------------------
I did a test on my laptop (2 cores, 4 virtual cores via hyper-threading)
using StreamingUpdateSolrServer with 1,2,4,8, and 16 threads.
The documents were just 1M unique ids, and the throughput ranged anywhere from 60,000 docs per second, to 67000.
The timing of different runs varied up to 6% for the same number of threads on the client, but I couldn't detect a difference when the ReadWriteLock was switched to fair mode.
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (SOLR-2342) Lock starvation can cause commit to
never run when many clients are adding docs
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990554#comment-12990554 ]
Michael McCandless commented on SOLR-2342:
------------------------------------------
Fixing the client threads to not commit in the end, so no client is committing, and then turning on Solr's auto-commit, still hits the starvation.
> Lock starvation can cause commit to never run when many clients are adding docs
> -------------------------------------------------------------------------------
>
> Key: SOLR-2342
> URL: https://issues.apache.org/jira/browse/SOLR-2342
> Project: Solr
> Issue Type: Bug
> Components: update
> Reporter: Michael McCandless
> Priority: Minor
>
> I have a stress test, where 100 clients add 100 1MB docs and then call commit in the end. It's a falldown test (try to make Solr fall down) and nowhere near "actual" usage.
> But, after some initial commits that succeed, I'm seeing later commits always time out (client side timeout @ 10 minutes). Watching Solr's logging, no commit ever runs.
> Looking at the stack traces in the threads, this is not deadlock: the add/update calls are running, and new segments are being flushed to the index.
> Digging in the code a bit, we use ReentrantReadWriteLock, with add/update acquiring the readLock and commit acquiring the writeLock. But, according to the jdocs, the writeLock isn't given any priority over the readLock (unless you set fairness, which we don't). So I think this explains the starvation?
> However, this is not a real world use case (most apps would/should call commit less often, and from on client). Also, we could set fairness, but it seems to have some performance penalty, and I'm not sure we should penalize the "normal" case for this unusual one. EG see here (thanks Mark): http://www.javaspecialists.eu/archive/Issue165.html.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org