You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2011/06/01 09:33:47 UTC
[jira] [Created] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Prevent IW#close and cut over to IW#commit
------------------------------------------
Key: SOLR-2565
URL: https://issues.apache.org/jira/browse/SOLR-2565
Project: Solr
Issue Type: Improvement
Components: update
Affects Versions: 4.0
Reporter: Simon Willnauer
Fix For: 4.0
Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075993#comment-13075993 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Yeah, sorry - it's open as a reminder for me to make that changes note (or at least evaluate if something should be done) and do the wiki documentation. I'll try and do that tomorrow if I can and get this closed.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042761#comment-13042761 ]
Robert Muir commented on SOLR-2565:
-----------------------------------
As far as tests go on this one, personally I am happy.
It would be good if people could review the branch (its really not a lot of code), and maybe give some opinions on the TODOs such as indexwriter setting changes on core reloads, etc.
I think it would be better to merge this into trunk sooner than later and iterate in trunk myself.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096716#comment-13096716 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
I'd been considering a default auto-soft commit by time.
If you think every minute is a good hard commit upper bound due to the transaction log, my initial thoughts are:
Auto commit by time
hard commit: 60 seconds
soft commit: 1 second
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057425#comment-13057425 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Okay - looks like this applies and tests pass.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090331#comment-13090331 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Hossman has been working on a new test and it has picked up a further issue along the lines of the one Vadim brought up - if using time based auto commit for both hard and soft commits, the soft commits won't happen when triggered by deletes.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087405#comment-13087405 ]
Uwe Schindler commented on SOLR-2565:
-------------------------------------
This is the thread about the slowness: [http://www.lucidimagination.com/search/document/a7dcdd77c8b0d14d/solr_tests_in_trunk_sloooooooooooooooooooooooooow]
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated SOLR-2565:
---------------------------
Attachment: SOLR-2565__HuperDuperAutoCommitTest.patch
As mark mentioned, i started looking into a better overall meme for testing autocommit -- i could never reproduce any of the failures charlie cron was complaining about, but the tests didn't make any sense to me anyway.
path demonstrates a new overall approach, using more detailed monitor of events -- not just "the most recent event" but all of them in a queue of (rough) timestamps.
at the moment one of the tests has an @Ignore:nocommit because of the delete bug that miller mentioned, but it would be helpful to know if people who were seeing problems running AutoCOmmitTest.testSoftAndHardCommitMaxTime (ie: charlie cron and simon) could try this patch out and see if it works better for them.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated SOLR-2565:
------------------------------
Attachment: SOLR-2565.patch
here is a patch, showing all changes from the throwaway branch.
I'm going on vacation for a week, i hope this thing is committed... if not I may just commit it myself the day I come back.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059587#comment-13059587 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
I've still got to put a note in changes about how you should reload SolrCores after this change.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054755#comment-13054755 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Threats, threats...
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Vadim Kisselmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084810#comment-13084810 ]
Vadim Kisselmann commented on SOLR-2565:
----------------------------------------
Hi Mark,
thanks for the quick reply and fix.
I test it tomorrow at work and report.
Regards Vadim
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087453#comment-13087453 ]
Hoss Man commented on SOLR-2565:
--------------------------------
Summary of IRC discussion (as i remember it) that didn't make it to the mailing list...
* Use and rmuir found that they could consistently reproduce "slow" and "fast" behavior in some tests using fixed seeds...
** {{ant test -Dtestcase=SolrExampleBinaryTest -Dtests.seed=1:1:1}} ... always slow
** {{ant test -Dtestcase=SolrExampleBinaryTest -Dtests.seed=2:2:2}} ... always fast
* for Uwe and sarowe (on windows), slow ment "taking so long they finally just had to give up and kill the JVM"
* For rmuir on his linux box, slow vs. fast was ~1-10 seconds vs 100 seconds
* For hossman on my linux box, results were roughly consistent with rmuir, but slightly slower since i have a crappier box
* with the path Uwe commited in r1159448 rmuir, uwe, and sarowe all agreed the tests were "fast" again
* testUnicode (where the majority of the slowness is seen) is one of the few test methods in these slow classes that uses random testing in a loop, so the compounded speed issues there are likely from some cumulative slow down that is typically only done rarely in other tests (likely "commit" given the nature of the commit that seemed to cause the problem)
* Jenkin's is using a test multiplier of "3" which is specifically used in testUnicode to pick the number of iterations, so that obviously compounds the effects of this slowdown in jenkins -- but makes me highly suspicious about why Uwe and sarowe were seeing so much longer test times on their windows machines then rmuir or i were seeing on linux (since they weren't using any special test multiplier)
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-2565) Prevent IW#close and cut
over to IW#commit
Posted by "Vadim Kisselmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083094#comment-13083094 ]
Vadim Kisselmann edited comment on SOLR-2565 at 8/11/11 2:39 PM:
-----------------------------------------------------------------
I tested the nightly build #1595 with an new patch (2565), but NRT doesn't work in my case.
I index 10 docs/sec, it takes 1-30sec. to see the results.
same behavior when i update an existing document.
My addedDate is an timestamp (default="NOW"). In worst case i can see what the document which i indexed is already more when 30
seconds in my index, but i can't see it.
My Settings:
<autoCommit>
<maxDocs>1000</maxDocs>
<maxTime>60000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxDocs>1</maxDocs>
<maxTime>1000</maxTime>
</autoSoftCommit>
Are my settings wrong or need you more details?
Should i use the coldSearcher (default=false)? Or set maxWarmingSearchers higher than 2?
UPDATE:
If i only use autoSoftCommit and uncomment autoCommit it works.
But i should use the "hard" autoCommit, right?
Mark said yes:
http://www.lucidimagination.com/blog/2011/07/11/benchmarking-the-new-solr-‘near-realtime’-improvements/
was (Author: ldb):
I tested the nightly build #1595 with an new patch (2565), but NRT doesn't work in my case.
I index 10 docs/sec, it takes 1-30sec. to see the results.
same behavior when i update an existing document.
My addedDate is an timestamp (default="NOW"). In worst case i can see what the document which i indexed is already more when 30
seconds in my index, but i can't see it.
My Settings:
<autoCommit>
<maxDocs>1000</maxDocs>
<maxTime>60000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxDocs>1</maxDocs>
<maxTime>1000</maxTime>
</autoSoftCommit>
Are my settings wrong or need you more details?
Should i use the coldSearcher (default=false)? Or set maxWarmingSearchers higher than 2?
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Vadim Kisselmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083094#comment-13083094 ]
Vadim Kisselmann commented on SOLR-2565:
----------------------------------------
I tested the nightly build #1595 with an new patch (2565), but NRT doesn't work in my case.
I index 10 docs/sec, it takes 1-30sec. to see the results.
same behavior when i update an existing document.
My addedDate is an timestamp (default="NOW"). In worst case i can see what the document which i indexed is already more when 30
seconds in my index, but i can't see it.
My Settings:
<autoCommit>
<maxDocs>1000</maxDocs>
<maxTime>60000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxDocs>1</maxDocs>
<maxTime>1000</maxTime>
</autoSoftCommit>
Are my settings wrong or need you more details?
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042048#comment-13042048 ]
Robert Muir commented on SOLR-2565:
-----------------------------------
I merged the initial commit from SOLR-2279 (which enables MockDirectoryWrapper for solr tests/tracking file handles)... no issues found from the branch.
I'm gonna look now at some other general ways we can beef up the test coverage in Solr, especially by reusing what we have done in Lucene.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057318#comment-13057318 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
If this patch applies, I'll commit it.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Vadim Kisselmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085072#comment-13085072 ]
Vadim Kisselmann commented on SOLR-2565:
----------------------------------------
Hi Mark,
it works, thanks!
The docs are equally available.
Now i'm fighting with "Internal Server Errors" caused by such queries:
http://localhost:8080/solr/clustering?wt=ruby&rows=40&start=0&fl=description+pubDate&sort=pubDate+desc&q=london
It's this Issue: https://issues.apache.org/jira/browse/LUCENE-2208?focusedCommentId=13082315#comment-13082315
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084494#comment-13084494 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Checked it real quick before I head out - it's a mistake in trying to not hard and soft commit at the same time. It's just not correct.
I have a test case and fix - I'll post a patch tomorrow.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141651#comment-13141651 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
bq. that test now fails for me due to what looks like a timing issue
Two of the tests in hossmans new AutoCommit test where failing for me on my linux box - i tweaked both to pass and also tested them on my mac. I've also reenabled the AutoCommit test class, but removed the tests involving soft commit - they will be tested with this new method in the new test class instead - later we can move the original auto commit test methods to the new style.
I'll commit soon.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller reassigned SOLR-2565:
---------------------------------
Assignee: Mark Miller
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2565:
------------------------------
Attachment: SOLR-2565.patch
here is the fix and a test, I'll commit soon.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075980#comment-13075980 ]
Jason Rutherglen commented on SOLR-2565:
----------------------------------------
This issue says committed in the comments, however it's status is: "Unresolved"?
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081967#comment-13081967 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
+1 - makes sense, looks right.
Without the patch, your stress test fails for me on my new linux box in 6-12 seconds regularly. With the patch I just ran it for half an hour.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated SOLR-2565:
--------------------------------
Attachment: SOLR-2565-revert.patch
This is the "minimal revert". With this patch applied, the SolrJ tests are running fast again.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090463#comment-13090463 ]
Robert Muir commented on SOLR-2565:
-----------------------------------
I ran the original patch (with the @ignore still enabled) 100x each on my fast and slow machine.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096346#comment-13096346 ]
Yonik Seeley commented on SOLR-2565:
------------------------------------
So now that commits no longer block adds, we should revisit what the best defaults are.
The transaction logging in SOLR-2700 needs to keep track of uncommitted documents - hence if it doesn't affect performance too much, we should probably commit more often some how. Autocommit based on number of documents doesn't work well across the broad spectrum of users (what value would work well for twitter indexers and book indexers). A size-based approach would probably work best, but we don't have that. Maybe a time based approach? That would limit the transaction log size to the number of documents indexable in a given time period, which should be roughly proportional to the document size. I guess such a time period should be somewhere between 10 and 60 seconds? A lot of data can be indexed in 60 sec, and the goal is to limit the transaction log size while not impacting performance too much due to increased commit frequency.
The other issue is soft commits... should we configure a soft commitWithin by default (prob within the range of 1-10 sec)?
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086755#comment-13086755 ]
Jason Rutherglen commented on SOLR-2565:
----------------------------------------
Can this one be backported to 3.x? It would probably be fairly useful for people to use now?
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090412#comment-13090412 ]
Robert Muir commented on SOLR-2565:
-----------------------------------
Looks cool, I'm beasting this test on a couple of machines now...
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller resolved SOLR-2565.
-------------------------------
Resolution: Fixed
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley updated SOLR-2565:
-------------------------------
Attachment: SOLR-2565.patch
With the following patch, TestRealTimeGet ran without errors (I stopped it after it ran for over an hour).
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057829#comment-13057829 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Committed - there is still some wiki work to do.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087410#comment-13087410 ]
Uwe Schindler commented on SOLR-2565:
-------------------------------------
I committed the partial revert in revision: 1159448
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081775#comment-13081775 ]
Yonik Seeley commented on SOLR-2565:
------------------------------------
I think the removal of all of the synchronization in the update handler leads to a race condition on reopen.
Basically, the order that we start a reopen in different threads does not have to correspond to the order that the reopen finishes, or the order that the new searcher is registered. This can mean temporarily lost documents - a request can add a document + repoen and due to scheduling have an older reopen registered last in a different thread (meaning the document won't be visible).
So it looks like we need to synchronize to ensure that only one thread reopens at a time.
As far as I can tell, I don't think that this synchronization needs to extend to the IW.commit() for a hard commit.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084487#comment-13084487 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Thanks Vadim - I'll look at this tomorrow.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086986#comment-13086986 ]
Michael McCandless commented on SOLR-2565:
------------------------------------------
+1 to backport; this is a serious limitation in Solr.
ElasticSearch has been using this approach (NRT & commit, not IW.close) for quite some time already.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated SOLR-2565:
--------------------------------
Attachment: dump.txt
I get the same thread dumps, sometimes also with a MockDirectoryFactory.maybeYield (seldom), and this one.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, dump.txt, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102088#comment-13102088 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
Further work to be followed up in other issues - I've put in the basic doc on the wiki around this.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081804#comment-13081804 ]
Yonik Seeley commented on SOLR-2565:
------------------------------------
My TestRealTimeGet currently fails after a few minutes... it may be due to this (hopefully). I'll try fixing.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096778#comment-13096778 ]
Mark Miller commented on SOLR-2565:
-----------------------------------
bq. there could be a race issue w/ autocommit
I wouldn't be surprised - the AutoCommit tracker has never been 100% thread safe code. Though I did make some improvements, I didn't address everything I saw. I think it's generally been benign stuff though, so could be something else too.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-2565) Prevent IW#close and cut
over to IW#commit
Posted by "Vadim Kisselmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083094#comment-13083094 ]
Vadim Kisselmann edited comment on SOLR-2565 at 8/11/11 2:16 PM:
-----------------------------------------------------------------
I tested the nightly build #1595 with an new patch (2565), but NRT doesn't work in my case.
I index 10 docs/sec, it takes 1-30sec. to see the results.
same behavior when i update an existing document.
My addedDate is an timestamp (default="NOW"). In worst case i can see what the document which i indexed is already more when 30
seconds in my index, but i can't see it.
My Settings:
<autoCommit>
<maxDocs>1000</maxDocs>
<maxTime>60000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxDocs>1</maxDocs>
<maxTime>1000</maxTime>
</autoSoftCommit>
Are my settings wrong or need you more details?
Should i use the coldSearcher (default=false)? Or set maxWarmingSearchers higher than 2?
was (Author: ldb):
I tested the nightly build #1595 with an new patch (2565), but NRT doesn't work in my case.
I index 10 docs/sec, it takes 1-30sec. to see the results.
same behavior when i update an existing document.
My addedDate is an timestamp (default="NOW"). In worst case i can see what the document which i indexed is already more when 30
seconds in my index, but i can't see it.
My Settings:
<autoCommit>
<maxDocs>1000</maxDocs>
<maxTime>60000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxDocs>1</maxDocs>
<maxTime>1000</maxTime>
</autoSoftCommit>
Are my settings wrong or need you more details?
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096774#comment-13096774 ]
Yonik Seeley commented on SOLR-2565:
------------------------------------
I did some performance tests w/ autoCommit=60000 (60 sec) vs no commits at all (10M small docs)
6 runs for each configuration, times in seconds.
no commits (not even at the end): 202,222,217,206,203,196 average=207
autoCommit=60000: 201,206,210,226,220,201 average=210
So committing every 60 seconds actually came out faster (probably due to chance because of the high variability).
Aside: there could be a race issue w/ autocommit. I was checking the segments_n files to verify that commits were actually taking place periodically. On a new index, the first file is segments_1. All of the tests ran around 3 minutes 30 seconds... which should have meant 4 more commits, or a segments_5 file after 4 minutes. I only checked a few times, but one time I saw a segments_6, and two other times I saw a segments_9 file. This suggests that too many commits are happening for some reason.
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-2565:
------------------------------
Attachment: fix+hossmans-test.patch
a fix for the delete issue and hossmans test with the test method that was ignored not ignored anymore.
that test now fails for me due to what looks like a timing issue
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565-revert.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, SOLR-2565__HuperDuperAutoCommitTest.patch, dump.txt, fix+hossmans-test.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2565) Prevent IW#close and cut over to
IW#commit
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated SOLR-2565:
------------------------------
Attachment: slowtests.txt
thread dumps of slow tests
> Prevent IW#close and cut over to IW#commit
> ------------------------------------------
>
> Key: SOLR-2565
> URL: https://issues.apache.org/jira/browse/SOLR-2565
> Project: Solr
> Issue Type: Improvement
> Components: update
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2565.patch, SOLR-2565.patch, SOLR-2565.patch, slowtests.txt
>
>
> Spinnoff from SOLR-2193. We already have a branch to work on this issue here https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193
> The main goal here is to prevent solr from closing the IW and use IW#commit instead. AFAIK the main issues here are:
> The update handler needs an overhaul.
> A few goals I think we might want to look at:
> 1. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing:
> 2. Stop closing the IndexWriter and start using commit (still lazy IW init though).
> 3. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
> 4. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index.
> Eventually this is a preparation for NRT support in Solr which I will create a followup issue for.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org