You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Per Steffensen (Created) (JIRA)" <ji...@apache.org> on 2012/02/29 13:17:57 UTC
[jira] [Created] (SOLR-3178) Versioning - optimistic locking
Versioning - optimistic locking
-------------------------------
Key: SOLR-3178
URL: https://issues.apache.org/jira/browse/SOLR-3178
Project: Solr
Issue Type: New Feature
Components: update
Affects Versions: 3.5
Environment: All
Reporter: Per Steffensen
Assignee: Per Steffensen
Fix For: 4.0
In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265686#comment-13265686 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
"Fire and forget" will also be much cooler with a real async interface. We will provide that very soon - see SOLR-3383.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258131#comment-13258131 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
And SOLR-3384, which is kinda independent but still cool. Im not sure we will deal with this issue for now though
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258226#comment-13258226 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
bq. I will change my strategy and deliver a rough patch as soon as the merge has finished
That's great, I'm sure you'll find more success with breaking it up into many smaller issues - different committers will be able to help on different things as their interest & time allow.
I'd really like to try and get the bare essentials of optimistic locking in soon, so the sooner you can show us a draft patch for just that piece so we can decide what approach to go with, the better! There's no need to wait to merge up to trunk to show us how your approach differs.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271273#comment-13271273 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
Sorry to see that you chose to commit a different patch for this one, making me believe that you are not going to take my patch in. I really dont understand why, because I think my solution is at least as good or better in any aspect, and besides that it is more complete, backward compatible and solves a lot of other issues (SOLR-3173, SOLR-3382 and SOLR-3428) already - can I ask you to tell me why you made that choice? Maybe what is wrong with our patch since it is not taken in, what we need to change etc?
Please also elaborate on your plans wrt the other issues
* Using unique key constraints to protect against overwriting existing data, when intent is to just insert a new document (kinda what SORL-3173 was thought to be about) - DocumentAlreadyExists error
* Even though an document update fails, still try updating the rest of the documents in the same update request, and therefore the feature of being able to report several errors back to the client linking each error to a specific document in the request (basically SOLR-3382)
* Nicely "typed" errors so that you as a client can detect which kind of error occurred in an easier way than guessing from http status code and textual error messages (also SOLR-3382)
* The SOLR-3428 bug
I need to know what to tell my product owners wrt how long we need to run on our own version of Solr (containing our patches etc.) in our high-throughput and high-concurrency pilot-prod system and therefore also for how long we need to keep merging changes from Solr trunk in? And when we can get to contribute additional code - mostly stability/endurance-related bug fixes. We already gave you the fix for SOLR-3437 but some of our additional fixes are kinda dependent on some of the stuff in SOLR-3173_3178_3382_3428_plus.patch. Do you have any plans revealing if we have to wait
* Forever? (= our patch is not going to be committed)
* Weeks/Days? (= you are working on understanding our patch and expect to commit later)
* Or?
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267463#comment-13267463 ]
Per Steffensen edited comment on SOLR-3178 at 5/3/12 2:32 PM:
--------------------------------------------------------------
Find attached SOLR-3173_3178_3382_3428_plus.patch
The patch fits on top of trunk revision 1332666 and is ready for commit.
Since last patch (SOLR_3173_3178_3382_plus.patch) I have made the following:
* Implemented test ClassicConsistencyHybridUpdateSemanticsSolrCloudTest verifying that partial errors are propagated correctly in a cloud (ZK) setup, when an update document is not sent directly to the Solr instance running the leader of the slice where the document is to be stored, and the Solr instance therefore has to forward the document to the leader, and when the leader forwards to replica. This made me discover the problems discribed in SOLR-3428, which I had to fix in order to make the test green.
* Implemented tests (in JsonLoaderTest and CSVRequestHandlerTest) verifying that you can send part references as claimed on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics when you are sending JSON or CSV requests
* Documents for which a partial error occur on leader will not be forwarded to replica.
* Partial errors are now catched and collected for the entire request in DistributedUpdateProcessor, but the actual version-check etc is still performed in DirectUpdateHandler2 (because that is an per-document-level task)
* Added Apache License 2.0 text to all new files
* Added class-level-JavaDoc to all new classes
* Removed a few unimportant changes
I will be leaving for an extended weekend now, and will not be available again before wedensday next week. I will probably read and answer comments from time to time until saturday night, though.
I really really really hope to return to the great news about the patch having been committed. I believe it is now really ready - well worked through features, well covered by tests and all existing plus new tests are green. Great progress, maybe not perfection, but we can shape the last edges later if we find any. Hope you will take the opportunity to commit before too long, now that you have a patch based on a fairly new revision, to avoid having to many conflicts to solve.
Just to avoid any mistakes - the patch covers SOLR-3173, SOLR-3178, SOLR-3382 and SOLR-3428.
Regards, Per Steffensen
was (Author: steff1193):
Find attached SOLR-3173_3178_3382_3428_plus.patch
The patch fits on top of trunk revision 1332666 and is ready for commit.
Since last patch (SOLR_3173_3178_3382_plus.patch) I have made the following:
* Implemented test ClassicConsistencyHybridUpdateSemanticsSolrCloudTest verifying that partial errors are propagated correctly in a cloud (ZK) setup, when an update document is not sent directly to the Solr instance running the leader of the slice where the document is to be stored, and the Solr instance therefore has to forward the document to the leader, and when the leader forwards to replica. This made me discover the problems discribed in SOLR-3428, which I had to fix in order to make the test green.
* Implemented tests (in JsonLoaderTest and CSVRequestHandlerTest) verifying that you can send part references as claimed on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics when you are sending JSON or CSV requests
* Documents for which a partial error occur on leader will not be forwarded to replica.
* Partial errors are now catched and collected for the entire request in DistributedUpdateProcessor, but the actual version-check etc is still performed in DirectUpdateHandler2 (because that is an per-document-level task)
* Added Apache License 2.0 text to all new files
* Added class-level-JavaDoc to all new classes
* Removed a few unimportant changes
I will be leaving for an extended weekend now, and will not be available again before wedensday next week. I will probably read and answer comments from time to time until saturday night, though.
I really really really hope to return to the great news about the patch having been committed. I believe it is now really ready - well worked through features, well covered by tests and all existing plus new tests are green. Great progress, maybe not perfection, but we can shape the last edges later if we find any. Hope you will take the opportunity to commit before too long, now that you have a patch based on a fairly new revision, to avoid having to many conflicts to solve.
Just to avoid any mistakes - the patch covers SORL-3173 , SORL-3178 , SOLR-3382 and SOLR-3428.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257682#comment-13257682 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
bq. it is hard to push too many small partly-done features.
As committers it's the opposite - it's sometimes hard to consume patches that do a lot of different things.
bq. It works with multi-document update request, providing fully detailed feedback to the client about which document-updates failed (and why) and which succeeded.
That's great! But it's also a separate feature that we've needed for a while (and I think there has been a JIRA issue open for it for a while).
bq. Looking shortly at you patch, I belive, that if two threads in the server JVM overlaps in a way that is unfortunate enough, then it is possible that data will not be stored or will be overwritten without an exception being thrown to indicate that to the client.
I'm confident in the synchronization/concurrency part - it's just reusing what was put in place for SolrCloud to handle reordering of updates to replicas and is very well tested (see TestRealTimeGet). But please let us know if you see a problem with it, as that would mean a problem with SolrCloud today (even without this patch).
Some other additions that can be handled later that I see:
- SolrJ support for easier passing of _version_ on a delete, constants for _version_, etc
- Use of "1" as a generic "exists" version (i.e. update document only if it already exists)
- If one document in a batch fails, don't automatically abort, and provide info back about which docs succeeded and which failed (your first point).
That last one in particular needs some good discussion and design work and really deserves it's own issue.
Down to the specifics of this patch... it's very non-invasive to core, consisting of the following code block (once for add, once for delete):
{code}
if (versionOnUpdate != 0) {
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
long foundVersion = lastVersion == null ? -1 : lastVersion;
if ( versionOnUpdate == foundVersion || (versionOnUpdate < 0 && foundVersion < 0) ) {
// we're ok if versions match, or if both are negative (all missing docs are equal)
} else {
throw new SolrException(ErrorCode.CONFLICT, "version conflict for " + cmd.getPrintableId() + " expected=" + versionOnUpdate + " actual=" + foundVersion);
}
}
{code}
Having this current improvement committed in no way blocks any future improvements you may come up with (including deleting the code quoted above and using whatever method you have come up with for checking the versions), and it even uses the same API (or a subset of it) via _version_ and 409. Progress, not perfection!
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley updated SOLR-3178:
-------------------------------
Attachment: SOLR-3178.patch
Here's a patch that seems to work fine.
There's really no extra input-API - all you do is supply a _version_ in the document (or as a parameter on the URL for a delete) and you get back a 409 if it's incorrect.
I'll probably commit this soon and we can incrementally improve by adding better support to SolrJ (I'm not sure there's an easy way to specify the version for a delete from solrJ).
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258224#comment-13258224 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
Mark Miller, its noted and I will certainly try to ajust behaviour for future work on Solr. Thanks for helping me understand the Apache process better.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257625#comment-13257625 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
I really hope you will give us a chance to contribute the solution to this SOLR-3178 and SOLR-3173. We have a solution that Im sure you will think is great - a solution that we will send to you within the next couple of weeks.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265768#comment-13265768 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
bq. since I offer the "semanticsMode" property in updatehandler
Seems like this should be up to the client... why wouldn't update semantics be handled on a per-request basis?
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258109#comment-13258109 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
And SOLR-3383
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257984#comment-13257984 ]
Mark Miller commented on SOLR-3178:
-----------------------------------
Per Steffensen, can you attach whatever you have, even if it's in rough form, so that the two approaches can be compared?
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257664#comment-13257664 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
I dont care about credit :-)
It is just that I put myself on as Assigned to indicate that I was working on it. And I have been - with a pause of a few weeks because my product owner prioritized me to look at something different (money talkes - you know :-) ). I am soon ready to provide a full solution so why not wait for that. My solution is a little bit different than yours (just had a very short peek at you patch) but basically it is doing the same thing, but there is no need for both you and my solution. I really hope you will wait until you see my contribution - it is much more complete than what you have done until now, and we are already using it with success in a highly concurrent pilot-production environment.
My contribution solves at least a few problems that yours doesnt seem to solve (yet):
- It works with multi-document update request, providing fully detailed feedback to the client about which document-updates failed (and why) and which succeeded. I believe your solution will just result in a 409 HTTP response if just one of many documents in the same request failed, leaving no information to the user about which of the other documents in the request where successfully handled, and if more of the documents (comming after the on that did fail) would also have failed.
- It works (have been tested by heavily concurrent tests) in highly concurrent environments. Looking shortly at you patch, I belive, that if two threads in the server JVM overlaps in a way that is unfortunate enough, then it is possible that data will not be stored or will be overwritten without an exception being thrown to indicate that to the client.
I understand your point about doing this incrementally, and I have done that internally. But since I am not a Solr committer it is hard to push too many small partly-done features. If I had been a committer you would have been able to follow the progress of my work more closely. But fact is, that I have a full and well tested solution that I would like to provide as a contribution very soon. I really hope you will wait for that, before committing different a partly done solution to the same problems. Thanks a lot!
I would much rather want you to carefully read http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and provide feedback and comments. I will deliver everything described there (and more) very soon.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Comment Edited] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291668#comment-13291668 ]
Per Steffensen edited comment on SOLR-3178 at 6/8/12 11:13 AM:
---------------------------------------------------------------
{quote} Regarding error handling, I tracked down the original issue: SOLR-445 {quote}
Yes, SOLR-445 is solved by my patch - the nice way. On certain kinds of errors (PartialError subclasses) during the handling of a particular document in a nultidocument/batch-update the processing of subsequent documents will continue. The client will receive a response describing all errors (wrapped in PartialErrors) that happend during the processing of the entire update-request (multidocument/batch). Please have a look at http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics#Multi_document_updates
{quote} It's just a guess, but I think it unlikely any committers would feel comfortable tackling this big patch, or even have time to understand all of the different aspects. They may agree with some parts but disagree with other parts {quote}
Of course that is up to you, but I believe Solr has a problem being a real Open Source project receiving contributions from many semi-related organistions around the world, if you do not "trust your test suite". Basically when taking in a patch a committer do not need to understand everything down to every detail. It should be enough (if you "trust your test suite") to
* Verify that all existing tests are still green - and havnt been hacked
* Verify that all new tests seem to be meaningfull and covering the features described in the corresponding Jira (and in my case the associated Wiki page), indicating that the new features are usefull and well tested (in order to be able to "trust the test suite" will reveal if future commits will ruin this new feature)
* Scan through the new code to see if it is breaking any design principals etc., and in general if it seems to be doing the right thing the right way
As long as a patch does not break any existing functionality, and seems to bring nice new functionality (you should be able to see that from the added tests) a patch cannot be that harmfull - you can always refactor if you realize that you "disagree with some parts". It all depends on "trusting your test suite". Dont you agree, in principle at least?
Regards, Per Steffensen
was (Author: steff1193):
{quote} Regarding error handling, I tracked down the original issue: SOLR-445 {quote}
Yes, SOLR-445 is solved by my patch - the nice way. On certain kinds of errors (PartialError subclasses) during the handling of a particular document in a nultidocument/batch-update the processing of subsequent documents will continue. The client will receive a response describing all errors (wrapped in PartialErrors) that happend during the processing of the entire update-request (multidocument/batch). Please have a look at http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics#Multi_document_updates
{quote} It's just a guess, but I think it unlikely any committers would feel comfortable tackling this big patch, or even have time to understand all of the different aspects. They may agree with some parts but disagree with other parts {quote}
Of course that is up to you, but I believe Solr has a problem being a real Open Source project receiving contributions from many semi-related organistions around the world, if you do not "trust your test suite". Basically when taking in a patch a committer do not need to understand everything down to every detail. It should be enough (if you "trust your test suite") to
* Verify that all existing tests are still green - and havnt been hacked
* Verify that all new tests seem to be meaningfull and covering the features described in the corresponding Jira (and in my case the associated Wiki page), indicating that the new features are usefull and well tested
* Scan through the new code to see if it is breaking any design principals etc., and in general if it seems to be doing the right thing the right way
As long as a patch does not break any existing functionality, and seems to bring nice new functionality (you should be able to see that from the added tests) a patch cannot be that harmfull - you can always refactor if you realize that you "disagree with some parts". It all depends on "trusting your test suite". Dont you agree, in principle at least?
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265685#comment-13265685 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
Agree, people have different preferences on how to use Solr, and therefore a.o. how the exact semantics of update should be. My solution is also better prepared for that, since I offer the "semanticsMode" property in updatehandler (DirectUpdateHandler2) where you can state if you want strict versioning ("consistency" mode) or the good old semantics ("classic" mode) or a mix ("classic-consistency-mode" mode). Guess we will always need to do leader-first-then-replica I cases where we want to do strict versioning control, but in "classic" mode (or a new future mode) we could certainly consider doing leader-and-replica-in-parallel. Will provide new patch today or tomorrow.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285695#comment-13285695 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
I had missed this last update during the crazy week of the lucene revolution conference.
It's just a guess, but I think it unlikely any committers would feel comfortable tackling this big patch, or even have time to understand all of the different aspects. They may agree with some parts but disagree with other parts. Small, tightly focused patches that tackle a single aspect are much more likely to get attention.
Just speaking for myself, when I get time to look at optimistic locking stuff again, I'll probably browse your latest patch to see what improvements I can cherrypick (assuming no one else gets around to making some smaller patches first).
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258186#comment-13258186 ]
Mark Miller commented on SOLR-3178:
-----------------------------------
bq. I will change my strategy and deliver a rough patch as soon as the merge has finished
+1 - the Apache model likes dev to happen in the open as much as possible in piecemeal - it's more digestible, and helps ensure a lot of work is not done in a direction that others may not end up agreeing with. You don't want to dump a ton of finely polished work that has been fully documented and then find the community has other intentions.
Even a patch that does not compile has a lot of value! Just specify it's early state and limitations - but it lets us know the general direction you are taking, allows early feedback, and helps any committers digest the work.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265601#comment-13265601 ]
Mark Miller commented on SOLR-3178:
-----------------------------------
Yeah, it would be nice to offer a wide range of indexing options - I'd really like to see us offer down to "fire and forget" - the add waits for no response and adds are done locally and remotely in parallel. AFAIK MongoDB actually does this by default - I don't think we should do that, but I'm interested in offering a range from that up to full guarantees.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257004#comment-13257004 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
FYI, I've got a patch that seems to work fine... I'm just still finishing up some of the tests.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257630#comment-13257630 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
I think the best way to do this is really incrementally.
What's done so far should really be a strict sub-set of everything you were thinking of (since changed to use 409 as the error code), and is useful on it's own.
Don't worry... you'll get credit!
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254186#comment-13254186 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
One API issue is how to give error codes... looking at the HTTP list (http://en.wikipedia.org/wiki/List_of_HTTP_status_codes) there seem to be 2 that fit the bill:
409 Conflict
Indicates that the request could not be processed because of conflict in the request, such as an edit conflict.[2]
412 Precondition Failed
The server does not meet one of the preconditions that the requester put on the request.[2]
Looks like 409 is closer though.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated SOLR-3178:
------------------------------
Fix Version/s: (was: 4.0)
4.1
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.1
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR_3173_3178_3382_plus.patch, SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267463#comment-13267463 ]
Per Steffensen edited comment on SOLR-3178 at 5/3/12 2:32 PM:
--------------------------------------------------------------
Find attached SOLR-3173_3178_3382_3428_plus.patch
The patch fits on top of trunk revision 1332666 and is ready for commit.
Since last patch (SOLR_3173_3178_3382_plus.patch) I have made the following:
* Implemented test ClassicConsistencyHybridUpdateSemanticsSolrCloudTest verifying that partial errors are propagated correctly in a cloud (ZK) setup, when an update document is not sent directly to the Solr instance running the leader of the slice where the document is to be stored, and the Solr instance therefore has to forward the document to the leader, and when the leader forwards to replica. This made me discover the problems discribed in SOLR-3428, which I had to fix in order to make the test green.
* Implemented tests (in JsonLoaderTest and CSVRequestHandlerTest) verifying that you can send part references as claimed on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics when you are sending JSON or CSV requests
* Documents for which a partial error occur on leader will not be forwarded to replica.
* Partial errors are now catched and collected for the entire request in DistributedUpdateProcessor, but the actual version-check etc is still performed in DirectUpdateHandler2 (because that is an per-document-level task)
* Added Apache License 2.0 text to all new files
* Added class-level-JavaDoc to all new classes
* Removed a few unimportant changes
I will be leaving for an extended weekend now, and will not be available again before wedensday next week. I will probably read and answer comments from time to time until saturday night, though.
I really really really hope to return to the great news about the patch having been committed. I believe it is now really ready - well worked through features, well covered by tests and all existing plus new tests are green. Great progress, maybe not perfection, but we can shape the last edges later if we find any. Hope you will take the opportunity to commit before too long, now that you have a patch based on a fairly new revision, to avoid having to many conflicts to solve.
Just to avoid any mistakes - the patch covers SORL-3173 , SORL-3178 , SOLR-3382 and SOLR-3428.
Regards, Per Steffensen
was (Author: steff1193):
Find attached SOLR-3173_3178_3382_3428_plus.patch
The patch fits on top of trunk revision 1332666 and is ready for commit.
Since last patch (SOLR_3173_3178_3382_plus.patch) I have made the following:
* Implemented test ClassicConsistencyHybridUpdateSemanticsSolrCloudTest verifying that partial errors are propagated correctly in a cloud (ZK) setup, when an update document is not sent directly to the Solr instance running the leader of the slice where the document is to be stored, and the Solr instance therefore has to forward the document to the leader, and when the leader forwards to replica. This made me discover the problems discribed in SOLR-3428, which I had to fix in order to make the test green.
* Implemented tests (in JsonLoaderTest and CSVRequestHandlerTest) verifying that you can send part references as claimed on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics when you are sending JSON or CSV requests
* Documents for which a partial error occur on leader will not be forwarded to replica.
* Partial errors are now catched and collected for the entire request in DistributedUpdateProcessor, but the actual version-check etc is still performed in DirectUpdateHandler2 (because that is an per-document-level task)
* Added Apache License 2.0 text to all new files
* Added class-level-JavaDoc to all new classes
* Removed a few unimportant changes
I will be leaving for an extended weekend now, and will not be available again before wedensday next week. I will probably read and answer comments from time to time until saturday night, though.
I really really really hope to return to the great news about the patch having been committed. I believe it is now really ready - well worked through features, well covered by tests and all existing plus new tests are green. Great progress, maybe not perfection, but we can shape the last edges later if we find any. Hope you will take the opportunity to commit before too long, now that you have a patch based on a fairly new revision, to avoid having to many conflicts to solve.
Just to avoid any mistakes - the patch covers SORL-3173, SORL-3178, SOLR-3382 and SOLR-3428.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263514#comment-13263514 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
First of all. Thanks a lot for getting back to me, and taking the time to try to understand such a big patch covering several jira issues - Im still sorry about that.
{quote} I think I agree it would be nice to have more of the functionality consolidated to the update handler and not do quite so much "up the callstack"... of course the downside to this would be the context that the update processors have (namely they know when a request, which can contain multiple updates, starts and finishes). {quote}
I agree and understand about the different context, but failing/succeeding with PartialError (VersionConflict, DocumentAlreadyExists, DocumentDoesNotExist or WrongUsage) is a per-document-to-be-updated thing. Therefore I believe it does not matter in this case, or acutally, that might just be the argument that it belongs in UpdateHandler. Certainly it can be done in UpdateHandler.
{quote} Regardless - we are where we are right now... and doing the version check in the update handler presents some problems. For cloud, we want to start sending to replicas at the same time as indexing locally. {quote}
Im not sure I understand. As I read the code, we are not doing that as it is today. In DistributedUpdateProcessor.processAdd you first do the local add (as the leader) in versionAdd AND THEN, if you succeed, forwards to all replica (in the if (nodes != null) block). I believe it is a good strategy to first do update on leader and then, if success on leader, do update on replica concurrently (and asynchronously). So stick with that!
So doing version-check etc (when leader) in DirectUpdateHandler2 does not represent a problem. Errors occuring as a result of version-check etc should just (as any other error) result in update not forwarded to replica.
One problem in my patch is that DirectUpdateHandler2 does not notify (by throwing error or making sure dropCmd is true) DistributedUpdateProcessor to not send to replica in case of PartialError (VersionConflict, DocumentAlreadyExists, DOcumentDoesNotExists or WrongUsage). It will in the next upcomming patch.
{quote} We also want to check the version before sending to replicas since it would be easy to get into a scenario where a conditional update fails on some replicas and succeeds on others (due to the fact that updates can easily be seen by different replicas in a different order). {quote}
Agree about check version on leader before sending to replica. But that can equally well be done in "DistributedUpdateProcessor.versionAdd" (inside the locks) as you do, as it can be done in "DirectUpdateHandler2.addDoc" as I do. So with respect to that it doesnt matter if you take your patch or mine. Of course when I change my patch so that version-check etc errors in DirectUpdateHandler2 actually ensures that the update is not sent to replica.
The idea is that version-checks and uniqueKey-already-exists-checks etc, leading to PartialError, are only performed on leader, so replica will not fail with this kind of errors - and they shouldnt. Please note "cmd.isLeaderLogic()" in DirectUpdateHandler2.addDoc in the line
{code}
boolean getAndCheckAgainstExisting = semanticsMode.needToGetAndCheckAgainstExistingDocument(cmd) && cmd.isLeaderLogic();
{code}
Replica are supposed to be "dumb" in the way that they just index (without any checking) what was successfully indexed on the leader (and therefore forwarded to replica in the first place), and support getting updates in different orders by not indexing updates if they are "older" than an update they have already indexed.
{quote} So I think the best way forward is to commit the patch that does the version check in the distributed update processor, and then build off of that (for instance your more nuanced error messages and the set/get requestVersion on the update command). {quote}
I have to say that I do not agree. Believe I have argued why the problems you mention with my patch/approach is not true (when I make the small fix mentioned). Sure your patch is a small step forward (progress, not perfection), but building on top of that will put me in a situation where I have to make changes to my patch in order to get all the extra bennefits it provide (much more progresss, not perfection :-) ). Of course you can commit your patch, but I still believe the code in that patch should be deleted when adding my patch, and I still hope that will be the end result.
I will provide patch soon with the following changes
* PartialError in DirectUpdateHandler2 on leader will make DistributedUpdateProcessor not forward to replica (as with any other error on leader)
* Error propagation will work when client contact server that is not the leader of the document to be updated, and therefore forwards to leader, and therefore also have to propagate errors from that leader back to client. His has always worked for single errors, where you just abort the entire update on the first error, leaving no information to the client about what failed and what did not, but doesnt work now that you can have partial errors where some documents fail and other succeeds (on different servers) and you want to provide a detailed picture about it back to the client.
* Cleanup of a few things in the patch that should never have been included (survivals from early attempts rolled back on my side)
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257431#comment-13257431 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
We have been working on it ourselves (SOLR-3173 and SOLR-3178), and will really prefer that we get to contribute our changes. Please have a look at http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics - review comments are very very welcome. We will be ready to contribute the entire solution within the next couple of weeks. It is already running pilot-production inhouse and nicely handling 100+ mio inserts/updates per day.
We just decided to always use response code 400 (as you can see on the Wiki), but you suggestion about using other codes is good. I will hav a look at that.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257703#comment-13257703 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
Are there any technical objections to this patch?
It really piggybacks on all of the SolrCloud work done to pass around and check versions, so it's really small.
The API is as follows:
- If you provide a _version_ > 0 with an add or a delete, the update will fail with a 409 unless the provided _version_ matches exactly the latest in the index for the ID.
- If you provide a _version_ < 0 with an add or delete, the update will fail with a 409 if the ID exists in the index.
That's the whole scope of this patch, and I believe is compatible with the larger scope of what Per will provide in the future.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258064#comment-13258064 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
Mark Miller, we are working on a internal SVN branch where we started by talking in HEAD of apache solr trunk. From time to time we have merged new developments on apache solr trunk into our SVN branch. Right now we are in the process of doing that again, so that the patch we deliver will be a patch on top of a recent apache solr revision. Until that merge has finished I cannot attach our modifications unless you want them to fit on top of a pretty old (a month or so) revision of apache solr.
My intention was to deliver nicely finished (completely tested, documented etc) features to you, but based on what I have heard from you and Yonik the last couple of days, I will change my strategy and deliver a rough patch as soon as the merge has finished, and then leave the final polishing for a more complete patch a little bit later. I hope to be able to deliver my rough patch in the middle of next week.
Thanks a lot for you cooperation!
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257664#comment-13257664 ]
Per Steffensen edited comment on SOLR-3178 at 4/19/12 6:42 PM:
---------------------------------------------------------------
I dont care about credit :-)
It is just that I put myself on as Assigned to indicate that I was working on it. And I have been - with a pause of a few weeks because my product owner prioritized me to look at something different (money talkes - you know :-) ). I am soon ready to provide a full solution so why not wait for that. My solution is a little bit different than yours (just had a very short peek at your patch) but basically it is doing the same thing, but there is no need for both your and my solution. I really hope you will wait until you see my contribution - it is much more complete than what you have done until now, and we are already using it successfully in a highly concurrent pilot-production environment.
My contribution solves at least a few problems that yours doesnt seem to solve (yet):
- It (also) works for multi-document update request, providing fully detailed feedback to the client about which document-updates failed (and why) and which succeeded. I believe your solution will just result in a 409 HTTP response if just one of many documents in the same request failed, leaving no information to the user about which of the other documents in the request where successfully handled (and stored), and if more of the documents (comming after the on that did fail) would also have failed.
- It works (have been tested by heavily concurrent tests) in highly concurrent environments. Looking shortly at you patch, I belive, that if two threads in the server JVM overlaps in a way that is unfortunate enough, then it is possible that data will not be stored or will be overwritten without an error being sent back to the client indicating that.
- Feedback to client is even possible if using ConcurrentSolrServer, because it has been changed to provide a Future from "request" - a Future that will eventually provide the calling client with the result from the request.
I understand your point about doing this incrementally, and I have done that internally. But since I am not a Solr committer it is hard to push too many small partly-done features. If I had been a committer you would have been able to follow the progress of my work more closely. But fact is, that I have a full and well tested solution that I would like to provide as a contribution very soon. I really hope you will wait for that, before committing a different partly done solution to the same problems. Thanks a lot!
I would much rather want you to carefully read http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and provide feedback and comments. I will deliver everything described there (and more) very soon. And then tell me how to ajust remaining time on this issue :-)
Regards, Per Steffensen
was (Author: steff1193):
I dont care about credit :-)
It is just that I put myself on as Assigned to indicate that I was working on it. And I have been - with a pause of a few weeks because my product owner prioritized me to look at something different (money talkes - you know :-) ). I am soon ready to provide a full solution so why not wait for that. My solution is a little bit different than yours (just had a very short peek at you patch) but basically it is doing the same thing, but there is no need for both you and my solution. I really hope you will wait until you see my contribution - it is much more complete than what you have done until now, and we are already using it with success in a highly concurrent pilot-production environment.
My contribution solves at least a few problems that yours doesnt seem to solve (yet):
- It works with multi-document update request, providing fully detailed feedback to the client about which document-updates failed (and why) and which succeeded. I believe your solution will just result in a 409 HTTP response if just one of many documents in the same request failed, leaving no information to the user about which of the other documents in the request where successfully handled, and if more of the documents (comming after the on that did fail) would also have failed.
- It works (have been tested by heavily concurrent tests) in highly concurrent environments. Looking shortly at you patch, I belive, that if two threads in the server JVM overlaps in a way that is unfortunate enough, then it is possible that data will not be stored or will be overwritten without an exception being thrown to indicate that to the client.
I understand your point about doing this incrementally, and I have done that internally. But since I am not a Solr committer it is hard to push too many small partly-done features. If I had been a committer you would have been able to follow the progress of my work more closely. But fact is, that I have a full and well tested solution that I would like to provide as a contribution very soon. I really hope you will wait for that, before committing different a partly done solution to the same problems. Thanks a lot!
I would much rather want you to carefully read http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and provide feedback and comments. I will deliver everything described there (and more) very soon.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266385#comment-13266385 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
{quote} .bq since I offer the "semanticsMode" property in updatehandler {quote}
{quote} Seems like this should be up to the client... why wouldn't update semantics be handled on a per-request basis? {quote}
Maybe we misunderstand each other a little bit. I just agree that people want Solr to provide different kinds of semantics - some want strict version-check, some dont, some will have an fire-and-forget approach, some will not. I just ment to say that for some of those types of semantics you want server-side Solr to behave in a certain way. For that you probably need a configuration telling the server how to behave. I have introduced that for updates with the "semanticsMode" config on updatehandler.
I believe, when it comes to whether you want strict unique-key-constraint- and version-checks or not, it is probably something you want to configure server-side, so that you can control the possibilities of your clients. Not all Solr-based systems will have complete control over all its clients, and then it is nice on server-side to be able to say (using semanticsMode):
* consistency: I dont want you as a client to make updates to documents, without even having read the existing document first (and merged your updates into that), and I dont want you to be able to overwrite changes to that document that happend to be made between the time you read it and the time you provided your updated version. And I dont want you to incidentally overwrite an existing document, if your intent was to create a new one.
* classic: All updates just go right into Solr, but you risk overwriting exsisting stuff
* classic-concistency-hybrid: You can do both consistency and classic updates, be casefull when doing classic updates
I believe this kind of control belongs in a server-side config, because it is a way of "protecting" your data against badly written clients.
With respect to whether you want to do fire-and-forget or not, that should be up to the client, but it kinda is by definition - basically just ignore results. But fire-and-forget strategy on clients is really not worth much, if you have to wait for the server to reply when you do updates (like you have to do today). SOLR-3383 will provide a all-async client SolrJ API (Future-based all the way) so that you can really just "fire", and if you intent is to "forget" anyway, then you can move on immediately after your "fire" without waiting for the server to reply.
That is not the reason we made SOLR-3383 though. It is because we want to be able to get results back when using ConcurrentSolrServer, and that requires a Future-based request API.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Yonik Seeley (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257682#comment-13257682 ]
Yonik Seeley edited comment on SOLR-3178 at 4/19/12 7:14 PM:
-------------------------------------------------------------
bq. it is hard to push too many small partly-done features.
As committers it's the opposite - it's sometimes hard to consume patches that do a lot of different things.
bq. It works with multi-document update request, providing fully detailed feedback to the client about which document-updates failed (and why) and which succeeded.
That's great! But it's also a separate feature that we've needed for a while (and I think there has been a JIRA issue open for it for a while).
bq. Looking shortly at you patch, I belive, that if two threads in the server JVM overlaps in a way that is unfortunate enough, then it is possible that data will not be stored or will be overwritten without an exception being thrown to indicate that to the client.
I'm confident in the synchronization/concurrency part - it's just reusing what was put in place for SolrCloud to handle reordering of updates to replicas and is very well tested (see TestRealTimeGet). But please let us know if you see a problem with it, as that would mean a problem with SolrCloud today (even without this patch).
bq. Feedback to client is even possible if using ConcurrentSolrServer, because it has been changed to provide a Future from "request" - a Future that will eventually provide the calling client with the result from the request.
This sounds like another great feature! It really deserves it's own issue so people can more easily provide review/feedback and not have it coupled to this issue.
Some other additions that can be handled later that I see:
- SolrJ support for easier passing of _version_ on a delete, constants for _version_, etc
- Use of "1" as a generic "exists" version (i.e. update document only if it already exists)
- If one document in a batch fails, don't automatically abort, and provide info back about which docs succeeded and which failed (your first point).
That last one in particular needs some good discussion and design work and really deserves it's own issue.
Down to the specifics of this patch... it's very non-invasive to core, consisting of the following code block (once for add, once for delete):
{code}
if (versionOnUpdate != 0) {
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
long foundVersion = lastVersion == null ? -1 : lastVersion;
if ( versionOnUpdate == foundVersion || (versionOnUpdate < 0 && foundVersion < 0) ) {
// we're ok if versions match, or if both are negative (all missing docs are equal)
} else {
throw new SolrException(ErrorCode.CONFLICT, "version conflict for " + cmd.getPrintableId() + " expected=" + versionOnUpdate + " actual=" + foundVersion);
}
}
{code}
Having this current improvement committed in no way blocks any future improvements you may come up with (including deleting the code quoted above and using whatever method you have come up with for checking the versions), and it even uses the same API (or a subset of it) via _version_ and 409. Progress, not perfection!
was (Author: yseeley@gmail.com):
bq. it is hard to push too many small partly-done features.
As committers it's the opposite - it's sometimes hard to consume patches that do a lot of different things.
bq. It works with multi-document update request, providing fully detailed feedback to the client about which document-updates failed (and why) and which succeeded.
That's great! But it's also a separate feature that we've needed for a while (and I think there has been a JIRA issue open for it for a while).
bq. Looking shortly at you patch, I belive, that if two threads in the server JVM overlaps in a way that is unfortunate enough, then it is possible that data will not be stored or will be overwritten without an exception being thrown to indicate that to the client.
I'm confident in the synchronization/concurrency part - it's just reusing what was put in place for SolrCloud to handle reordering of updates to replicas and is very well tested (see TestRealTimeGet). But please let us know if you see a problem with it, as that would mean a problem with SolrCloud today (even without this patch).
Some other additions that can be handled later that I see:
- SolrJ support for easier passing of _version_ on a delete, constants for _version_, etc
- Use of "1" as a generic "exists" version (i.e. update document only if it already exists)
- If one document in a batch fails, don't automatically abort, and provide info back about which docs succeeded and which failed (your first point).
That last one in particular needs some good discussion and design work and really deserves it's own issue.
Down to the specifics of this patch... it's very non-invasive to core, consisting of the following code block (once for add, once for delete):
{code}
if (versionOnUpdate != 0) {
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
long foundVersion = lastVersion == null ? -1 : lastVersion;
if ( versionOnUpdate == foundVersion || (versionOnUpdate < 0 && foundVersion < 0) ) {
// we're ok if versions match, or if both are negative (all missing docs are equal)
} else {
throw new SolrException(ErrorCode.CONFLICT, "version conflict for " + cmd.getPrintableId() + " expected=" + versionOnUpdate + " actual=" + foundVersion);
}
}
{code}
Having this current improvement committed in no way blocks any future improvements you may come up with (including deleting the code quoted above and using whatever method you have come up with for checking the versions), and it even uses the same API (or a subset of it) via _version_ and 409. Progress, not perfection!
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Per Steffensen updated SOLR-3178:
---------------------------------
Attachment: SOLR-3173_3178_3382_3428_plus.patch
Find attached SOLR-3173_3178_3382_3428_plus.patch
The patch fits on top of trunk revision 1332666 and is ready for commit.
Since last patch (SOLR_3173_3178_3382_plus.patch) I have made the following:
* Implemented test ClassicConsistencyHybridUpdateSemanticsSolrCloudTest verifying that partial errors are propagated correctly in a cloud (ZK) setup, when an update document is not sent directly to the Solr instance running the leader of the slice where the document is to be stored, and the Solr instance therefore has to forward the document to the leader, and when the leader forwards to replica. This made me discover the problems discribed in SOLR-3428, which I had to fix in order to make the test green.
* Implemented tests (in JsonLoaderTest and CSVRequestHandlerTest) verifying that you can send part references as claimed on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics when you are sending JSON or CSV requests
* Documents for which a partial error occur on leader will not be forwarded to replica.
* Partial errors are now catched and collected for the entire request in DistributedUpdateProcessor, but the actual version-check etc is still performed in DirectUpdateHandler2 (because that is an per-document-level task)
* Added Apache License 2.0 text to all new files
* Added class-level-JavaDoc to all new classes
* Removed a few unimportant changes
I will be leaving for an extended weekend now, and will not be available again before wedensday next week. I will probably read and answer comments from time to time until saturday night, though.
I really really really hope to return to the great news about the patch having been committed. I believe it is now really ready - well worked through features, well covered by tests and all existing plus new tests are green. Great progress, maybe not perfection, but we can shape the last edges later if we find any. Hope you will take the opportunity to commit before too long, now that you have a patch based on a fairly new revision, to avoid having to many conflicts to solve.
Just to avoid any mistakes - the patch covers SORL-3173, SORL-3178, SOLR-3382 and SOLR-3428.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265686#comment-13265686 ]
Per Steffensen edited comment on SOLR-3178 at 5/1/12 6:47 AM:
--------------------------------------------------------------
"Fire and forget" will also be much cooler with a real async client interface. We will provide that very soon - see SOLR-3383.
was (Author: steff1193):
"Fire and forget" will also be much cooler with a real async interface. We will provide that very soon - see SOLR-3383.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257620#comment-13257620 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
I will have a look at the patch provided by you and merge any ideas in there that is not already in our solution into our solution, but please do not commit your patch. I really hope you will wait until we provide our patch which implements everything described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and more - e.g. SolrServers that return Future<NamedList<Object>> instead of just NamedList<Object>, which is needed to make the error propagation also described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics work for ConcurrentSolrServer. Everything in our contribution (the patch I deliver soon) will be fully covered by tests.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271273#comment-13271273 ]
Per Steffensen edited comment on SOLR-3178 at 5/10/12 6:05 PM:
---------------------------------------------------------------
Sad to see that you chose to commit a different patch for this one, making me believe that you are not going to take my patch in. I really dont understand why, because I think my solution is at least as good or better in any aspect, and besides that it is more complete, backward compatible and solves a lot of other issues (SOLR-3173, SOLR-3382 and SOLR-3428) already - can I ask you to tell me why you made that choice? Maybe what is wrong with our patch since it is not taken in, what we need to change etc?
Please also elaborate on your plans wrt the other issues
* Using unique key constraints to protect against overwriting existing data, when intent is to just insert a new document (kinda what SOLR-3173 was thought to be about) - DocumentAlreadyExists error
* Even though an document update fails, still try updating the rest of the documents in the same update request, and therefore the feature of being able to report several errors back to the client linking each error to a specific document in the request (basically SOLR-3382)
* Nicely "typed" errors so that you as a client can detect which kind of error occurred in an easier way than guessing from http status code and textual error messages (also SOLR-3382)
* The SOLR-3428 bug
I need to know what to tell my product owners wrt how long we need to run on our own version of Solr (containing our patches etc.) in our high-throughput and high-concurrency pilot-prod system and therefore also for how long we need to keep merging changes from Solr trunk in? And when we can get to contribute additional code - mostly stability/endurance-related bug fixes. We already gave you the fix for SOLR-3437 but some of our additional fixes are kinda dependent on some of the stuff in SOLR-3173_3178_3382_3428_plus.patch. Do you have any plans revealing if we have to wait
* Forever? (= our patch is not going to be committed)
* Weeks/Days? (= you are working on understanding our patch and expect to commit later)
* Or?
Regards, Per Steffensen
was (Author: steff1193):
Sorry to see that you chose to commit a different patch for this one, making me believe that you are not going to take my patch in. I really dont understand why, because I think my solution is at least as good or better in any aspect, and besides that it is more complete, backward compatible and solves a lot of other issues (SOLR-3173, SOLR-3382 and SOLR-3428) already - can I ask you to tell me why you made that choice? Maybe what is wrong with our patch since it is not taken in, what we need to change etc?
Please also elaborate on your plans wrt the other issues
* Using unique key constraints to protect against overwriting existing data, when intent is to just insert a new document (kinda what SORL-3173 was thought to be about) - DocumentAlreadyExists error
* Even though an document update fails, still try updating the rest of the documents in the same update request, and therefore the feature of being able to report several errors back to the client linking each error to a specific document in the request (basically SOLR-3382)
* Nicely "typed" errors so that you as a client can detect which kind of error occurred in an easier way than guessing from http status code and textual error messages (also SOLR-3382)
* The SOLR-3428 bug
I need to know what to tell my product owners wrt how long we need to run on our own version of Solr (containing our patches etc.) in our high-throughput and high-concurrency pilot-prod system and therefore also for how long we need to keep merging changes from Solr trunk in? And when we can get to contribute additional code - mostly stability/endurance-related bug fixes. We already gave you the fix for SOLR-3437 but some of our additional fixes are kinda dependent on some of the stuff in SOLR-3173_3178_3382_3428_plus.patch. Do you have any plans revealing if we have to wait
* Forever? (= our patch is not going to be committed)
* Weeks/Days? (= you are working on understanding our patch and expect to commit later)
* Or?
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258260#comment-13258260 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
Will supply patch early next week.
Guess I have broken (or will break it) the following "rule" on "How to contribute":
"Combine multiple issues into a single patch, especially if they are unrelated or only loosely related."
I have added the following to "How to contribute | Contributing your work" in order to state the philosophy described by Mark a little more clearly to future newbie contributers. Feel free to modify:
"Supply first patch as early as possible and updated patches as often as possible during your work. This helps the rest of the community and committers to start understanding, help shaping, commenting on etc. your work throughout the entire process. Supplying a patch does not necessarily mean that it is complete and ready to be committed, it might also just be a way of communicating your idea and progress."
Regards and have a nice weekend
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265095#comment-13265095 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
bq. > For cloud, we want to start sending to replicas at the same time as indexing locally.
bq. As I read the code, we are not doing that as it is today.
Right - I just realized the ambiguity of the phrasing. By "we want to start", I meant that we want to investigate/implement that option (to send to replicas concurrently with indexing locally).
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258083#comment-13258083 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
Yonik Seeley, I agree that some of the stuff that I have added deserves own issues, but it doesnt change the fact that we hope to contribute them all in one go. I understand that it would have been nicer if we had sent you many smaller contributions instead one big contribution. We will learn from that and do that for future contributions. I hope this time that you (the committers) will accept a big contribution.
I have created a separate issue for one of the things that is also going to be in our contribution - SOLR-3382
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260579#comment-13260579 ]
Per Steffensen edited comment on SOLR-3178 at 4/24/12 6:38 PM:
---------------------------------------------------------------
Find attached SOLR_3173_3178_3382_plus.patch which should fit on top of revision 1327417.
Patch includes:
- All our code-changes done during the work with SOLR-3173, SOLR-3178 and SOLR-3382. How it is supposed to work in more detail, is described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and is of course also expressed in included tests.
- Also includes
-- Misc clean-up
--- E.g. removing constants redundant OVERWRITE = "overwrite" so that only UpdateParams.OVERWRITE is left. You need very good arguments to ever use different "names" for the same thing (overwrite) among XML, JSON, HTTP request params etc, so it is kind of wrong to have the constant defined for each of those.
-- "Unimportant changes" - e.g.
--- Corrected spelling errors
--- Removed unnecessary imports
Not implemented yet
- Error propagation for redistributed updates. That is, the responses from the redistributed updates in DistributedUpdateProcessor using SolrCmdDistributor, have to be collected an merged into a combined response from "this" DistributedUpdateProcessor.
-- Both implementation and tests are missing.
- Tests verifying that updates using JSON and CSV requests as described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics also works as described. Especially transfer of partRefs in JSON and CSV requests.
- Tests verifying that semanticsMode "consistency" works as it is supposed to - just like ClassicConsistencyHybridXXXTests tests this for "classic-consistency-hybrid" semanticsMode and like ClassicUpdateSemanticsTest shortly tests it when semanticsMode is explicitly set to "classic" ("classic" is the default so this mode is already tested a lot using all the others tests in the project)
Other stuff not done yet
- Add Apache 2.0 Licence to all new files
- Check correct indenting
- JavaDoc for public methods
was (Author: steff1193):
Find attached SOLR_3173_3178_3382_plus.patch which should fit on top of revision 1327417.
Patch includes:
- All our code-changes done during the work with SOLR-3173, SOLR-3178 and SOLR-3382. How it is supposed to work in more detail, is described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and is of course also expressed in included tests.
- Also includes
-- Misc clean-up
--- E.g. removing constants redundant OVERWRITE = "overwrite" so that only UpdateParams.OVERWRITE is left. You need very good arguments to ever use different "names" for the same thing (overwrite) among XML, JSON, HTTP request params etc, so it is kind of wrong to have the constant defined for each of those.
-- "Unimportant changes" - e.g.
--- Corrected spelling errors
--- Removed unnecessary imports
Not implemented yet
- Error propagation for redistributed updates. That is, the responses from the redistributed updates in DistributedUpdateProcessor using SolrCmdDistributor, have to be collected an merged into a combined response from "this" DistributedUpdateProcessor.
-- Both implementation and tests are missing.
- Tests verifying that updates using JSON and CSV requests as described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics also works as described. Especially transfer of partRefs in JSON and CSV requests.
- Tests verifying that semanticsMode "consistency" works as it is supposed to - just like ClassicConsistencyHybridXXXTests tests this for "classic-consistency-hybrid" semanticsMode and like ClassicUpdateSemanticsTest shortly tests it when semanticsMode is explicitly set to "classic" ("classic" is the default so this mode is already tested a lot using all the others tests in the project)
Other stuff not done yet
- Add Apache 2.0 Licence to all new files
- Check correct indenting
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263514#comment-13263514 ]
Per Steffensen edited comment on SOLR-3178 at 4/27/12 3:13 PM:
---------------------------------------------------------------
First of all. Thanks a lot for getting back to me, and taking the time to try to understand such a big patch covering several jira issues - Im still sorry about that.
{quote} I think I agree it would be nice to have more of the functionality consolidated to the update handler and not do quite so much "up the callstack"... of course the downside to this would be the context that the update processors have (namely they know when a request, which can contain multiple updates, starts and finishes). {quote}
I agree and understand about the different context, but failing/succeeding with PartialError (VersionConflict, DocumentAlreadyExists, DocumentDoesNotExist or WrongUsage) is a per-document-to-be-updated thing. Therefore I believe it does not matter in this case, or acutally, that might just be the argument that it belongs in UpdateHandler. Certainly it can be done in UpdateHandler.
{quote} Regardless - we are where we are right now... and doing the version check in the update handler presents some problems. For cloud, we want to start sending to replicas at the same time as indexing locally. {quote}
Im not sure I understand. As I read the code, we are not doing that as it is today. In DistributedUpdateProcessor.processAdd you first do the local add (as the leader) in versionAdd AND THEN, if you succeed, forwards to all replica (in the if (nodes <> null) block). I believe it is a good strategy to first do update on leader and then, if success on leader, do update on replica concurrently (and asynchronously). So stick with that!
So doing version-check etc (when leader) in DirectUpdateHandler2 does not represent a problem. Errors occuring as a result of version-check etc should just (as any other error) result in update not forwarded to replica.
One problem in my patch is that DirectUpdateHandler2 does not notify (by throwing error or making sure dropCmd is true) DistributedUpdateProcessor to not send to replica in case of PartialError (VersionConflict, DocumentAlreadyExists, DOcumentDoesNotExists or WrongUsage). It will in the next upcomming patch.
{quote} We also want to check the version before sending to replicas since it would be easy to get into a scenario where a conditional update fails on some replicas and succeeds on others (due to the fact that updates can easily be seen by different replicas in a different order). {quote}
Agree about check version on leader before sending to replica. But that can equally well be done in "DistributedUpdateProcessor.versionAdd" (inside the locks) as you do, as it can be done in "DirectUpdateHandler2.addDoc" as I do. So with respect to that it doesnt matter if you take your patch or mine. Of course when I change my patch so that version-check etc errors in DirectUpdateHandler2 actually ensures that the update is not sent to replica.
The idea is that version-checks and uniqueKey-already-exists-checks etc, leading to PartialError, are only performed on leader, so replica will not fail with this kind of errors - and they shouldnt. Please note "cmd.isLeaderLogic()" in DirectUpdateHandler2.addDoc in the line
{code}
boolean getAndCheckAgainstExisting = semanticsMode.needToGetAndCheckAgainstExistingDocument(cmd) && cmd.isLeaderLogic();
{code}
Replica are supposed to be "dumb" in the way that they just index (without any checking) what was successfully indexed on the leader (and therefore forwarded to replica in the first place), and support getting updates in different orders by not indexing updates if they are "older" than an update they have already indexed.
{quote} So I think the best way forward is to commit the patch that does the version check in the distributed update processor, and then build off of that (for instance your more nuanced error messages and the set/get requestVersion on the update command). {quote}
I have to say that I do not agree. Believe I have argued why the problems you mention with my patch/approach is not true (when I make the small fix mentioned). Sure your patch is a small step forward (progress, not perfection), but building on top of that will put me in a situation where I have to make changes to my patch in order to get all the extra bennefits it provide (much more progresss, not perfection :-) ). Of course you can commit your patch, but I still believe the code in that patch should be deleted when adding my patch, and I still hope that will be the end result.
I will provide patch soon with the following changes
* PartialError in DirectUpdateHandler2 on leader will make DistributedUpdateProcessor not forward to replica (as with any other error on leader)
* Error propagation will work when client contact server that is not the leader of the document to be updated, and therefore forwards to leader, and therefore also have to propagate errors from that leader back to client. His has always worked for single errors, where you just abort the entire update on the first error, leaving no information to the client about what failed and what did not, but doesnt work now that you can have partial errors where some documents fail and other succeeds (on different servers) and you want to provide a detailed picture about it back to the client.
* Cleanup of a few things in the patch that should never have been included (survivals from early attempts rolled back on my side)
was (Author: steff1193):
First of all. Thanks a lot for getting back to me, and taking the time to try to understand such a big patch covering several jira issues - Im still sorry about that.
{quote} I think I agree it would be nice to have more of the functionality consolidated to the update handler and not do quite so much "up the callstack"... of course the downside to this would be the context that the update processors have (namely they know when a request, which can contain multiple updates, starts and finishes). {quote}
I agree and understand about the different context, but failing/succeeding with PartialError (VersionConflict, DocumentAlreadyExists, DocumentDoesNotExist or WrongUsage) is a per-document-to-be-updated thing. Therefore I believe it does not matter in this case, or acutally, that might just be the argument that it belongs in UpdateHandler. Certainly it can be done in UpdateHandler.
{quote} Regardless - we are where we are right now... and doing the version check in the update handler presents some problems. For cloud, we want to start sending to replicas at the same time as indexing locally. {quote}
Im not sure I understand. As I read the code, we are not doing that as it is today. In DistributedUpdateProcessor.processAdd you first do the local add (as the leader) in versionAdd AND THEN, if you succeed, forwards to all replica (in the if (nodes != null) block). I believe it is a good strategy to first do update on leader and then, if success on leader, do update on replica concurrently (and asynchronously). So stick with that!
So doing version-check etc (when leader) in DirectUpdateHandler2 does not represent a problem. Errors occuring as a result of version-check etc should just (as any other error) result in update not forwarded to replica.
One problem in my patch is that DirectUpdateHandler2 does not notify (by throwing error or making sure dropCmd is true) DistributedUpdateProcessor to not send to replica in case of PartialError (VersionConflict, DocumentAlreadyExists, DOcumentDoesNotExists or WrongUsage). It will in the next upcomming patch.
{quote} We also want to check the version before sending to replicas since it would be easy to get into a scenario where a conditional update fails on some replicas and succeeds on others (due to the fact that updates can easily be seen by different replicas in a different order). {quote}
Agree about check version on leader before sending to replica. But that can equally well be done in "DistributedUpdateProcessor.versionAdd" (inside the locks) as you do, as it can be done in "DirectUpdateHandler2.addDoc" as I do. So with respect to that it doesnt matter if you take your patch or mine. Of course when I change my patch so that version-check etc errors in DirectUpdateHandler2 actually ensures that the update is not sent to replica.
The idea is that version-checks and uniqueKey-already-exists-checks etc, leading to PartialError, are only performed on leader, so replica will not fail with this kind of errors - and they shouldnt. Please note "cmd.isLeaderLogic()" in DirectUpdateHandler2.addDoc in the line
{code}
boolean getAndCheckAgainstExisting = semanticsMode.needToGetAndCheckAgainstExistingDocument(cmd) && cmd.isLeaderLogic();
{code}
Replica are supposed to be "dumb" in the way that they just index (without any checking) what was successfully indexed on the leader (and therefore forwarded to replica in the first place), and support getting updates in different orders by not indexing updates if they are "older" than an update they have already indexed.
{quote} So I think the best way forward is to commit the patch that does the version check in the distributed update processor, and then build off of that (for instance your more nuanced error messages and the set/get requestVersion on the update command). {quote}
I have to say that I do not agree. Believe I have argued why the problems you mention with my patch/approach is not true (when I make the small fix mentioned). Sure your patch is a small step forward (progress, not perfection), but building on top of that will put me in a situation where I have to make changes to my patch in order to get all the extra bennefits it provide (much more progresss, not perfection :-) ). Of course you can commit your patch, but I still believe the code in that patch should be deleted when adding my patch, and I still hope that will be the end result.
I will provide patch soon with the following changes
* PartialError in DirectUpdateHandler2 on leader will make DistributedUpdateProcessor not forward to replica (as with any other error on leader)
* Error propagation will work when client contact server that is not the leader of the document to be updated, and therefore forwards to leader, and therefore also have to propagate errors from that leader back to client. His has always worked for single errors, where you just abort the entire update on the first error, leaving no information to the client about what failed and what did not, but doesnt work now that you can have partial errors where some documents fail and other succeeds (on different servers) and you want to provide a detailed picture about it back to the client.
* Cleanup of a few things in the patch that should never have been included (survivals from early attempts rolled back on my side)
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263263#comment-13263263 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
Thanks for the patch Per!
I think I agree it would be nice to have more of the functionality consolidated to the update handler and not do quite so much "up the callstack"... of course the downside to this would be the context that the update processors have (namely they know when a request, which can contain multiple updates, starts and finishes).
Regardless - we are where we are right now... and doing the version check in the update handler presents some problems. For cloud, we want to start sending to replicas at the same time as indexing locally. We also want to check the version before sending to replicas since it would be easy to get into a scenario where a conditional update fails on some replicas and succeeds on others (due to the fact that updates can easily be seen by different replicas in a different order).
So I think the best way forward is to commit the patch that does the version check in the distributed update processor, and then build off of that (for instance your more nuanced error messages and the set/get requestVersion on the update command).
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3178) Versioning - optimistic
locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266385#comment-13266385 ]
Per Steffensen edited comment on SOLR-3178 at 5/2/12 6:39 AM:
--------------------------------------------------------------
bq. {quote} since I offer the "semanticsMode" property in updatehandler {quote}
{quote} Seems like this should be up to the client... why wouldn't update semantics be handled on a per-request basis? {quote}
Maybe we misunderstand each other a little bit. I just agree that people want Solr to provide different kinds of semantics - some want strict version-check, some dont, some will have an fire-and-forget approach, some will not. I just ment to say that for some of those types of semantics you want server-side Solr to behave in a certain way. For that you probably need a configuration telling the server how to behave. I have introduced that for updates with the "semanticsMode" config on updatehandler.
I believe, when it comes to whether you want strict unique-key-constraint- and version-checks or not, it is probably something you want to configure server-side, so that you can control the possibilities of your clients. Not all Solr-based systems will have complete control over all its clients, and then it is nice on server-side to be able to say (using semanticsMode):
* consistency: I dont want you as a client to make updates to documents, without even having read the existing document first (and merged your updates into that), and I dont want you to be able to overwrite changes to that document that happend to be made between the time you read it and the time you provided your updated version. And I dont want you to incidentally overwrite an existing document, if your intent was to create a new one.
* classic: All updates just go right into Solr, but you risk overwriting exsisting stuff
* classic-concistency-hybrid: You can do both consistency and classic updates, be casefull when doing classic updates
I believe this kind of control belongs in a server-side config, because it is a way of "protecting" your data against badly written clients.
With respect to whether you want to do fire-and-forget or not, that should be up to the client, but it kinda is by definition - basically just ignore results. But fire-and-forget strategy on clients is really not worth much, if you have to wait for the server to reply when you do updates (like you have to do today). SOLR-3383 will provide a all-async client SolrJ API (Future-based all the way) so that you can really just "fire", and if you intent is to "forget" anyway, then you can move on immediately after your "fire" without waiting for the server to reply.
That is not the reason we made SOLR-3383 though. It is because we want to be able to get results back when using ConcurrentSolrServer, and that requires a Future-based request API.
Regards, Per Steffensen
was (Author: steff1193):
{quote} .bq since I offer the "semanticsMode" property in updatehandler {quote}
{quote} Seems like this should be up to the client... why wouldn't update semantics be handled on a per-request basis? {quote}
Maybe we misunderstand each other a little bit. I just agree that people want Solr to provide different kinds of semantics - some want strict version-check, some dont, some will have an fire-and-forget approach, some will not. I just ment to say that for some of those types of semantics you want server-side Solr to behave in a certain way. For that you probably need a configuration telling the server how to behave. I have introduced that for updates with the "semanticsMode" config on updatehandler.
I believe, when it comes to whether you want strict unique-key-constraint- and version-checks or not, it is probably something you want to configure server-side, so that you can control the possibilities of your clients. Not all Solr-based systems will have complete control over all its clients, and then it is nice on server-side to be able to say (using semanticsMode):
* consistency: I dont want you as a client to make updates to documents, without even having read the existing document first (and merged your updates into that), and I dont want you to be able to overwrite changes to that document that happend to be made between the time you read it and the time you provided your updated version. And I dont want you to incidentally overwrite an existing document, if your intent was to create a new one.
* classic: All updates just go right into Solr, but you risk overwriting exsisting stuff
* classic-concistency-hybrid: You can do both consistency and classic updates, be casefull when doing classic updates
I believe this kind of control belongs in a server-side config, because it is a way of "protecting" your data against badly written clients.
With respect to whether you want to do fire-and-forget or not, that should be up to the client, but it kinda is by definition - basically just ignore results. But fire-and-forget strategy on clients is really not worth much, if you have to wait for the server to reply when you do updates (like you have to do today). SOLR-3383 will provide a all-async client SolrJ API (Future-based all the way) so that you can really just "fire", and if you intent is to "forget" anyway, then you can move on immediately after your "fire" without waiting for the server to reply.
That is not the reason we made SOLR-3383 though. It is because we want to be able to get results back when using ConcurrentSolrServer, and that requires a Future-based request API.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226443#comment-13226443 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
More detailed descriptions of the added features in SOLR-3173 and SOLR-3178 here: http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291052#comment-13291052 ]
Yonik Seeley commented on SOLR-3178:
------------------------------------
Regarding error handling, I tracked down the original issue: SOLR-445
bq. BTW, I saw that you committed some "inc" stuff on this JIRA.
An accident... I was traveling and got the JIRA number wrong on the commit. The JIRA for updateable documents (inc/add/set, etc) is
SOLR-139
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290868#comment-13290868 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
{quote} It's just a guess, but I think it unlikely any committers would feel comfortable tackling this big patch, or even have time to understand all of the different aspects. They may agree with some parts but disagree with other parts. Small, tightly focused patches that tackle a single aspect are much more likely to get attention.{quote}
I understand, but things are very related, so its hard for me to go back and divide into smaller patches and feed them one by one to you. Things are related because "optimistic locking"-errors and "unique key constraints"-errors are not worth much if exact information about what went wrong does not go back to the client (therefore adding the feature of sending typed errors back to the client is needed). And the two kinds on new errros are the first "single document"-related errors in Solr, and therefore it, for the first time in history of Solr, is very nice to have the ability to keep processing documents in a multi-document-update-request even though one document-update fails with one of those errors (therefore multi-error responses with references on each error to the document it relates to is needed). And backward compability is just always nice, and it this case required, IMHO
{quote} Just speaking for myself, when I get time to look at optimistic locking stuff again, I'll probably browse your latest patch to see what improvements I can cherrypick (assuming no one else gets around to making some smaller patches first). {quote}
I would really like that, and I will be available for any help you need. Or just make me a committer, and you will live happily ever after :-) I am very experienced doing this kind of stuff.
BTW, I saw that you committed some "inc" stuff on this JIRA. As I understand it, it is a new feature where you are able to tell server-side Solr to add to a field, instead of doing it yourself on client side. Server-side Solr can then do it while having the lock on the document, and therefore no optimistic locking is needed in principle - but only if your updates use "inc"s only. I think it deserves its own JIRA, because it doesnt have anything to do with optimistic locking, except that you do not need optimistic locking if your updates use "inc"s only. It is probably not realistic that any real world application would ever be able to do its updates based on "inc"s alone, so in isolation I think the feature is kinda useless. But if you generalize on the feature it could be great. Generalize in a way where you can have any update-semantic performed on your document server-side (while having the document lock), then utilizing this feature heavily would enable you to avoid using optimistic locking and that might be prefered in some situations - just even another reason while backward compability on optimistic locking is nice, so that you can have it disabled (even though you would like to have a full-featured Cloud-setup with updateLog, _version_ field etc.) if you are not going to use it.
I imagine that you can install document(-field)-manipulating primitives on server-side Solr and associate them with a names (e.g. associate the name "inc" with a piece of code adding to a field, "multiply" with a piece of code multiplying to a field, "string-concat" with a piece of code that concats to fields into a third field, etc) and then have pairs of document-field and document(-field)-manipulating-ref in your updates (e.g. send updates saying "update document A manipulate fields inc(field1,13), multiply(field2,4), string-concat(field3,field4,field5)"). But as long as "inc"s is the only such manipulate-on-server-side-while-having-the-lock-primitive it is kinda useless. The code you associate with primitives ("inc", "multiply", "string-concat" etc) could just be a small piece of Java code, and the association needs to go on in solrconfig.xml. Or if you are really cool (and you trust your clients are not doing SQL-injection-ish hacks on you) you can allow clients to install primitives on-demand, either as part of update-requests that use those primitives or as separate primitive-registration-request that you do before you start using a primitive. Such request-carried primitives could be in the form of Groovy code, Ruby code, JavaScript code, or whatever the JVM is able to compile and use on-demand.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Per Steffensen updated SOLR-3178:
---------------------------------
Attachment: SOLR-3173_3178_3382_3428_plus.patch
Updated patch SOLR-3173_3178_3382_3428_plus.patch to fit on top of Solr 4.0.0. Actually updated to fit on top of revision 1394844 of branch lucene_solr_4_0, but I believe this is the same as Solr 4.0.0.
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.1
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3173_3178_3382_3428_plus.patch, SOLR_3173_3178_3382_plus.patch, SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257439#comment-13257439 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
Changed the response codes to not always be 400. Comments and suggestions of changes are very welcome.
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291668#comment-13291668 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
{quote} Regarding error handling, I tracked down the original issue: SOLR-445 {quote}
Yes, SOLR-445 is solved by my patch - the nice way. On certain kinds of errors (PartialError subclasses) during the handling of a particular document in a nultidocument/batch-update the processing of subsequent documents will continue. The client will receive a response describing all errors (wrapped in PartialErrors) that happend during the processing of the entire update-request (multidocument/batch). Please have a look at http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics#Multi_document_updates
{quote} It's just a guess, but I think it unlikely any committers would feel comfortable tackling this big patch, or even have time to understand all of the different aspects. They may agree with some parts but disagree with other parts {quote}
Of course that is up to you, but I believe Solr has a problem being a real Open Source project receiving contributions from many semi-related organistions around the world, if you do not "trust your test suite". Basically when taking in a patch a committer do not need to understand everything down to every detail. It should be enough (if you "trust your test suite") to
* Verify that all existing tests are still green - and havnt been hacked
* Verify that all new tests seem to be meaningfull and covering the features described in the corresponding Jira (and in my case the associated Wiki page), indicating that the new features are usefull and well tested
* Scan through the new code to see if it is breaking any design principals etc., and in general if it seems to be doing the right thing the right way
As long as a patch does not break any existing functionality, and seems to bring nice new functionality (you should be able to see that from the added tests) a patch cannot be that harmfull - you can always refactor if you realize that you "disagree with some parts". It all depends on "trusting your test suite". Dont you agree, in principle at least?
Regards, Per Steffensen
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258838#comment-13258838 ]
Per Steffensen commented on SOLR-3178:
--------------------------------------
bq. {quote} it is hard to push too many small partly-done features. {quote}
bq. As committers it's the opposite - it's sometimes hard to consume patches that do a lot of different things.
I could imagine. Im sorry that it is possibly going to be a little bit like that this time. But I believe I will provide great stuff.
bq. {quote} It works with multi-document update request, providing fully detailed feedback to the client about which document-updates failed (and why) and which succeeded. {quote}
bq. That's great! But it's also a separate feature that we've needed for a while (and I think there has been a JIRA issue open for it for a while).
Yes I agree. See SOLR-3382. The solution I will provide is generic in the way that is supports giving feedback to the client about "parts" of a request that failed. The client will know that the "parts" which are not reported to have failed did succeed. Usefull for any kind of request where parts of it can fail and other parts can succeed. The client provide a "partRef" for all parts, and partial errors from the server each contain a "partRef" in order to let the client know which part(s) failed. I only use it for multi-document (or batch-updates as I see you call it in come places) updates (here each document is a "part"), but I guess it could also be used for other things like
- Reporting which operations of a multi-operation (e.g. one containing both adds, deletes, commits, etc) update that failed.
- Reporting if a single update partly failed (e.g. if using replication and the update was successfull on leader but not on one of more of the replicas)
- Etc.
bq. {quote} Looking shortly at you patch, I belive, that if two threads in the server JVM overlaps in a way that is unfortunate enough, then it is possible that data will not be stored or will be overwritten without an exception being thrown to indicate that to the client. {quote}
bq. I'm confident in the synchronization/concurrency part - it's just reusing what was put in place for SolrCloud to handle reordering of updates to replicas and is very well tested (see TestRealTimeGet). But please let us know if you see a problem with it, as that would mean a problem with SolrCloud today (even without this patch).
Looking closer at your code I would like to withdraw my concern about your solution. I didnt see that you are doing both "get existing version and check against provided version" and "store updated version" inside the same block governed by two sync-locks
- A readlock on a ReadWriteLock which is writelocked during deleteByQueries (your "vinfo.lockForUpdate"), protecting against concurent deletes (by query)
- A lock on the uniqueKey field (or a hash of it in order to be efficient) of the document (your "synchronized (bucket)"), protecting against concurrent updates and deletes (by id)
So basically I have repeated that locking-pattern in DirectUpdateHandler2 where I chose to place the "unique key constaint"- and "versioning"- check. Guess my locking can be removed if it is really necessary to have this kind of locking "this far up in the callstack" (in DistributedUpdateProcessor.versionAdd). For the purpose of "unique key constraint"- and "versioning"-check I believe it is only necessary to have this kind of locking around the actual updates in DirectUpdateHandler2 (when leader and not peer_sync) - and the smaller you can make synchronized blocks the better. But maybe you have reasons (related to SolrCloud) where you need the locking "this far up in the callstack".
But besides that fact that the locking I did in DirectUpdateHandler2 can be removed I really believe that the actual checking of "unique key constraint"- and "version"-violation logically belong in DirectUpdateHandler2, so I guess I will still try to convince you to take that part of my patch in.
Besides that, I believe your solution just implements what I call semantics-mode "classic-consistency-hybrid" on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics. I believe the "classic" mode and "consistency" mode are very nice to have in order to provide complete backward compabitility (classic mode) and a very strict consistency mode where it is not possible to "break the checks" by providing \_version\_ = 0 (consistency mode).
Finally I believe my solution is a little better with respect to "design" and "separation of concerns", by isolatating the rules and rule-checks in a separate class instead of just coding it directly inside DistributedUpdateProcessor which already deals with a lot of other concerns, but that is another discussion, and should not count as the deciding argument.
bq. {quote} Feedback to client is even possible if using ConcurrentSolrServer, because it has been changed to provide a Future from "request" - a Future that will eventually provide the calling client with the result from the request.{quote}
bq. This sounds like another great feature! It really deserves it's own issue so people can more easily provide review/feedback and not have it coupled to this issue.
Created the isolated issue - see SOLR-3383
{quote} Some other additions that can be handled later that I see:
SolrJ support for easier passing of version on a delete, constants for version, etc {quote}
Well basically I have moved VERSION_FIELD = "\_version\_" from VersionInfo to SolrInputDocument.
bq. Use of "1" as a generic "exists" version (i.e. update document only if it already exists)
Well I started out by wanting that feature (update only if exists, but no exact version check), but you couldnt see the good reason for that. Since then I realized that you are probably right and have removed that exact possibility - of course in classic and classic-consistency-hybrid modes you can still do updates without providing the exact version, but there is error if the document does not exist.
bq. If one document in a batch fails, don't automatically abort, and provide info back about which docs succeeded and which failed (your first point).
That is part of my solution - basically the thing making it necessary to add the ability to send many errors back to the client - SOLR-3382
bq. That last one in particular needs some good discussion and design work and really deserves it's own issue.
Agree. Go discuss and comment on SOLR-3382
{quote} Having this current improvement committed in no way blocks any future improvements you may come up with (including deleting the code quoted above and using whatever method you have come up with for checking the versions), and it even uses the same API (or a subset of it) via version and 409. Progress, not perfection! {quote}
Agree, so I guess you could go commit, but that would just make my merge-task a little bigger, so I hope you will hold your horses a few days more.
bq. Are there any technical objections to this patch?
Except for the things I mention here and there in this response, no. And those are all things that could be fixed later, going nicely hand in hand with your great philosophy of "progress, not perfection".
bq. It really piggybacks on all of the SolrCloud work done to pass around and check versions, so it's really small.
Yes it is small and beautiful
{quote} The API is as follows:
a) If you provide a version > 0 with an add or a delete, the update will fail with a 409 unless the provided version matches exactly the latest in the index for the ID.
b) If you provide a version < 0 with an add or delete, the update will fail with a 409 if the ID exists in the index.
{quote}
You always provide the same error (except the versions in the message). You do not distinguish between DocumentAlreadyExist, DocumentDoesNotExist and VersionConflict errors as I do. Of course the client would know that it is a DocumentAlreadyExists if he sent a negative version number and got 409 (at least as long as 409 is not used for other kind of conflicts that can also happen when version < 0) and that it is DocumentDoesNotExist or VersionConflict if he sent a positive version number and got 409 (at least ...). But he really doesnt know if it is DocumentDoesNotExist or VersionConflict unless he parses the error message, and that might be important to know because he might want to react differently - e.g.
- On DocumentDoesNotExist: Either do nothing or go do a create (version < 0) of the document
- On VersionConflict: Go refetch the document from Solr, merge in his changes (automatically or by the help from a user) and try update again.
Of course you also need to parse error-type in my solution to know exactly what the problem is, but you need no "rules" on client side (e.g. knowing that if you sent version > 0 and you got version-already-in-store < 0 back, it is DocumentDoesNotExist), and in SolrJ my solution automatically parses error-type and throw corresponding exceptions.
Ad b) Maybe you dont want an error if you try to delete a document which has already been deleted, but probably ok
bq. That's the whole scope of this patch, and I believe is compatible with the larger scope of what Per will provide in the future.
Agree!
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3178) Versioning - optimistic locking
Posted by "Per Steffensen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Per Steffensen updated SOLR-3178:
---------------------------------
Attachment: SOLR_3173_3178_3382_plus.patch
Find attached SOLR_3173_3178_3382_plus.patch which should fit on top of revision 1327417.
Patch includes:
- All our code-changes done during the work with SOLR-3173, SOLR-3178 and SOLR-3382. How it is supposed to work in more detail, is described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics and is of course also expressed in included tests.
- Also includes
-- Misc clean-up
--- E.g. removing constants redundant OVERWRITE = "overwrite" so that only UpdateParams.OVERWRITE is left. You need very good arguments to ever use different "names" for the same thing (overwrite) among XML, JSON, HTTP request params etc, so it is kind of wrong to have the constant defined for each of those.
-- "Unimportant changes" - e.g.
--- Corrected spelling errors
--- Removed unnecessary imports
Not implemented yet
- Error propagation for redistributed updates. That is, the responses from the redistributed updates in DistributedUpdateProcessor using SolrCmdDistributor, have to be collected an merged into a combined response from "this" DistributedUpdateProcessor.
-- Both implementation and tests are missing.
- Tests verifying that updates using JSON and CSV requests as described on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics also works as described. Especially transfer of partRefs in JSON and CSV requests.
- Tests verifying that semanticsMode "consistency" works as it is supposed to - just like ClassicConsistencyHybridXXXTests tests this for "classic-consistency-hybrid" semanticsMode and like ClassicUpdateSemanticsTest shortly tests it when semanticsMode is explicitly set to "classic" ("classic" is the default so this mode is already tested a lot using all the others tests in the project)
Other stuff not done yet
- Add Apache 2.0 Licence to all new files
- Check correct indenting
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent inserts, updates, deletes and queries in the entire lifetime of the index) instead of just a search index (first: everything indexed (in one thread), after: only queries), I would like Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide a version-number equal to "the version number I got when I fetched the existing document for update" plus one. If this provided version-number does not correspond to "the newest version-number of that document at the time of update" plus one, I will get a VersionConflict error. If it does correspond the document will be updated with the new one, so that "the newest version-number of that document" is NOW one higher than before the update. Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version number provided will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we need to be able to identify when a document "is the same" - that, per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org