You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mikhail Khludnev (JIRA)" <ji...@apache.org> on 2012/11/15 21:59:12 UTC
[jira] [Created] (SOLR-4085) Commit-free ExternalFileField
Mikhail Khludnev created SOLR-4085:
--------------------------------------
Summary: Commit-free ExternalFileField
Key: SOLR-4085
URL: https://issues.apache.org/jira/browse/SOLR-4085
Project: Solr
Issue Type: Improvement
Components: Schema and Analysis
Affects Versions: 4.1
Reporter: Mikhail Khludnev
Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Comment Edited] (SOLR-4085) Commit-free ExternalFileField
Posted by "Mikhail Khludnev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508499#comment-13508499 ]
Mikhail Khludnev edited comment on SOLR-4085 at 12/3/12 10:37 AM:
------------------------------------------------------------------
[~jpountz],
Your feedback is appreciated. To make this ticket even more valuable for community can we come through the particular points of confusing behavior, which you mention. Can you list them?
I also want to wait until [~romseygeek] leaves his feedback as a person who provided an interest to the subj.
Thank you, guys
was (Author: mkhludnev):
[~jpountz],
Your feedback is appreciated. To make this ticket even more valuable for community can we come through the particular points of confusing behavior, which you mention. Can you list them?
I also want to wait until [~romseygeek] leaves his feedback as a person who provided and interest to the subj.
Thank you, guys
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508713#comment-13508713 ]
Robert Muir commented on SOLR-4085:
-----------------------------------
I don't think this should be committed.
Then its only a matter of time before people open bugs, saying X or Y doesn't work and that some synchronization or hacky stuff should be added because of this, or even just its tests breaking
because of perfectly valid changes.
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508629#comment-13508629 ]
Adrien Grand commented on SOLR-4085:
------------------------------------
To me ExternalFileField is a convenient (but hacky) way to map documents to floats values in a static index. I understand that it may need to support updates (ReloadCacheRequestHandler) in order to seldom fix values.
Commit-free updates support however makes ExternalFileField look like the perfect candidate to support frequent document updates but I don't like the fact that it needs to read the whole file every time (with a java.io.Reader!) and resolve IDs against the top-level reader. Moreover the lack of atomicity makes it different from everything else we have in Lucene and Solr and I imagine people having a scoring function such as if (fieldA > 0) { return fieldB / fieldA } else { return 0 }: this might break badly if fieldA is an ExternalFileField and its valued is modified from 1 to 0 between the first and the second evaluation. I'm also a little worried about the problems you mentioned regarding concurrent reloads.
But on the other hand you didn't modify anything outside of ExternalFileField and FileFloatSource so it's perfectly fine with me if Alan or another committer decides this should be committed.
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Mikhail Khludnev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508703#comment-13508703 ]
Mikhail Khludnev commented on SOLR-4085:
----------------------------------------
- I agree. For me EFF is a workaround until we have field updates, or updateable DocValues, which is my dream btw.
- Then, issue which you mention is "delta reload". At contrast to the other ones it's the simplest and even trivial task. It's a plain java coding actually, there is no challenge in lucene internals here!
-- Straightforward -silly- approach is introducing "delta file" naming convention. i.e. external_rating_2012112812.txt will be "patched" by delta file external_rating_2012112812.delta etc.
-- But much more conscious approach is introduce ExternalDataAPI which will be backed onto current "file" impl and then can have few more efficient impls, supports delta updates. I think it can be the next step forward.
- if (fieldA > 0) { return fieldB / fieldA } else { return 0 } is an absolutely hit. Here I can only suggest to introduce zeroProofDivision function. There is no easy way to fix it in Lucene, because search request has no context there, at contrast to solr where it can be handled at QParser. We can try to achieve it by tricky query "warmer", which will dedupe queries before passing them into Lucene search
- concurrent reloads are not a EFF's sin. Someone can easily shoot the leg is start two dataimport processes in parallel, you know. Solr doesn't really protects from 'lifecycle' errors, like accidentally enabled "autocommit"
- don't you think that Resolving the issue with such strick verdict can scare away other committers. Can you ping someone as well you expressed you point of view? I wanted to involve [~yseeley@gmail.com] or [~hossman_lucene@fucit.org], but don't want hesitate to disturb them.
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-4085) Commit-free ExternalFileField
Posted by "Mikhail Khludnev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Khludnev updated SOLR-4085:
-----------------------------------
Attachment: SOLR-4085.patch
pls find patch with test coverage
Key design points:
- does not rely on file timestamp, just count number of reloads as a version
- version is held by every externalfilefield key.
- every FFValueSource receives current version timestam, and uses it for equality check - quite important for queryResultCache
-
- addresses stale cache entry problem. after field is reloaded new queries refers to it are not equal to the older ones that allows evict old entries from queryResultCache, etc
- doesn't covers atomicity problem when main query refers to the same field twice, one of them can be reloaded after another one is already in use
- support warming after commit. file is read and later provides cache hits;
- concurrent reloads (including lazy one) may lead to not really pleasant behavior. user should protect from such races externally anyway.
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Reopened] (SOLR-4085) Commit-free ExternalFileField
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrien Grand reopened SOLR-4085:
--------------------------------
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Mikhail Khludnev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508499#comment-13508499 ]
Mikhail Khludnev commented on SOLR-4085:
----------------------------------------
[~jpountz],
Your feedback is appreciated. To make this ticket even more valuable for community can we come through the particular points of confusing behavior, which you mention. Can you list them?
I also want to wait until [~romseygeek] leaves his feedback as a person who provided and interest to the subj.
Thank you, guys
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-4085) Commit-free ExternalFileField
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrien Grand resolved SOLR-4085.
--------------------------------
Resolution: Won't Fix
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Mikhail Khludnev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508970#comment-13508970 ]
Mikhail Khludnev commented on SOLR-4085:
----------------------------------------
[~rcmuir], I don't think this feature can be a fix for "bugs, saying X or Y doesn't work" it's about adding new ability. eg until updateable DocValues are delivered it's hardly possible to raise a bug "can't update docvalues", because it will be rejected 'by design'. Or I completely I misunderstood you.
[~yseeley@gmail.com] aha. thanks. Initially I had a test enforces atomicity, achieve it for QParsers via SolrQueryRequest.context, but failed to provide it for query cache warming. I'm not so self-assured to introduce treadlocal, but now it's clear how to do that.
[~jpountz] [~romseygeek] pls consider my point: Solr doesn't protect if somebody want to shoot his leg off. let's we have fairly valid updateXml:
{code}delQ *:* add-doc(1), add-doc(2),add-doc(3), commit {code} you can send this sequence *twice* in different requests - everything will be ok. But if these requests overlaps in time, you've got "an unpleasant behavior" user will see some intermediate state. If I got it right and Solr doesn't protect from externally initiated race (only DIH somehow does it), should we care about "intermediate state" and wasteful reloads in EFF.
Colleagues, please articulate your expectation for concurrent reload, and if I'd be able to conquer with them I handle "atomicity" also.
thanks
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Comment Edited] (SOLR-4085) Commit-free ExternalFileField
Posted by "Mikhail Khludnev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508703#comment-13508703 ]
Mikhail Khludnev edited comment on SOLR-4085 at 12/3/12 2:53 PM:
-----------------------------------------------------------------
- I agree. For me EFF is a workaround until we have field updates, or updateable DocValues, which is my dream btw.
- Then, issue which you mention is "delta reload". At contrast to the other ones it's the simplest and even trivial task. It's a plain java coding actually, there is no challenge in lucene internals here!
-- Straightforward -silly- approach is introducing "delta file" naming convention. i.e. external_rating_2012112812.txt will be "patched" by delta file external_rating_2012112812.delta etc.
-- But much more conscious approach is introduce ExternalDataAPI which will be backed onto current "file" impl and then can have few more efficient impls, supports delta updates. I think it can be the next step forward.
- if (fieldA > 0) { return fieldB / fieldA } else { return 0 } is an absolutely hit. Here I can only suggest to introduce zeroProofDivision function. There is no easy way to fix it in Lucene, because search request has no context there, at contrast to solr where it can be handled at QParser. We can try to achieve it by tricky query "warmer", which will dedupe queries before passing them into Lucene search
- concurrent reloads are not a EFF's sin. Someone can easily shoot the leg is start two dataimport processes in parallel, you know. Solr doesn't really protects from 'lifecycle' errors, like accidentally enabled "autocommit"
- don't you think that Resolving the issue with such strick verdict can scare away other committers. Can you ping someone as well you expressed you point of view? I wanted to involve [~yseeley@gmail.com] or [~hossman_lucene@fucit.org], but hesitated to disturb them.
was (Author: mkhludnev):
- I agree. For me EFF is a workaround until we have field updates, or updateable DocValues, which is my dream btw.
- Then, issue which you mention is "delta reload". At contrast to the other ones it's the simplest and even trivial task. It's a plain java coding actually, there is no challenge in lucene internals here!
-- Straightforward -silly- approach is introducing "delta file" naming convention. i.e. external_rating_2012112812.txt will be "patched" by delta file external_rating_2012112812.delta etc.
-- But much more conscious approach is introduce ExternalDataAPI which will be backed onto current "file" impl and then can have few more efficient impls, supports delta updates. I think it can be the next step forward.
- if (fieldA > 0) { return fieldB / fieldA } else { return 0 } is an absolutely hit. Here I can only suggest to introduce zeroProofDivision function. There is no easy way to fix it in Lucene, because search request has no context there, at contrast to solr where it can be handled at QParser. We can try to achieve it by tricky query "warmer", which will dedupe queries before passing them into Lucene search
- concurrent reloads are not a EFF's sin. Someone can easily shoot the leg is start two dataimport processes in parallel, you know. Solr doesn't really protects from 'lifecycle' errors, like accidentally enabled "autocommit"
- don't you think that Resolving the issue with such strick verdict can scare away other committers. Can you ping someone as well you expressed you point of view? I wanted to involve [~yseeley@gmail.com] or [~hossman_lucene@fucit.org], but don't want hesitate to disturb them.
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508770#comment-13508770 ]
Yonik Seeley commented on SOLR-4085:
------------------------------------
Hi Mikhail,
Although I don't have time to review this patch right now, I think the general idea of this issue is fine.
We had a lack of atomicity with the date NOW in the past too, which was solved with a request-level thread local (SolrRequestInfo)
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Comment Edited] (SOLR-4085) Commit-free ExternalFileField
Posted by "Mikhail Khludnev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508703#comment-13508703 ]
Mikhail Khludnev edited comment on SOLR-4085 at 12/3/12 5:46 PM:
-----------------------------------------------------------------
- I agree. For me EFF is a workaround until we have field updates, or updateable DocValues, which is my dream btw.
- Then, issue which you mention is "delta reload". At contrast to the other ones it's the simplest and even trivial task. It's a plain java coding actually, there is no challenge in lucene internals here!
-- Straightforward -silly- approach is introducing "delta file" naming convention. i.e. external_rating_2012112812.txt will be "patched" by delta file external_rating_2012112812.delta etc.
-- But much more conscious approach is introduce ExternalDataAPI which will be backed onto current "file" impl and then can have few more efficient impls, supports delta updates. I think it can be the next step forward.
- if (fieldA > 0) { return fieldB / fieldA } else { return 0 } is an absolutely hit. Here I can only suggest to introduce zeroProofDivision function. There is no easy way to fix it in Lucene, because search request has no context there, at contrast to solr where it can be handled at QParser. We can try to achieve it by tricky query "warmer", which will dedupe queries before passing them into Lucene search
- concurrent reloads are not a EFF's sin. Someone can easily shoot the leg is start two dataimport processes in parallel, you know. Solr doesn't really protects from 'lifecycle' errors, like accidentally enabled "autocommit"
- don't you think that Resolving the issue with such strict verdict can scare away other committers. Can you ping someone as well you expressed you point of view? I wanted to involve [~yseeley@gmail.com] or [~hossman_lucene@fucit.org], but hesitated to disturb them.
was (Author: mkhludnev):
- I agree. For me EFF is a workaround until we have field updates, or updateable DocValues, which is my dream btw.
- Then, issue which you mention is "delta reload". At contrast to the other ones it's the simplest and even trivial task. It's a plain java coding actually, there is no challenge in lucene internals here!
-- Straightforward -silly- approach is introducing "delta file" naming convention. i.e. external_rating_2012112812.txt will be "patched" by delta file external_rating_2012112812.delta etc.
-- But much more conscious approach is introduce ExternalDataAPI which will be backed onto current "file" impl and then can have few more efficient impls, supports delta updates. I think it can be the next step forward.
- if (fieldA > 0) { return fieldB / fieldA } else { return 0 } is an absolutely hit. Here I can only suggest to introduce zeroProofDivision function. There is no easy way to fix it in Lucene, because search request has no context there, at contrast to solr where it can be handled at QParser. We can try to achieve it by tricky query "warmer", which will dedupe queries before passing them into Lucene search
- concurrent reloads are not a EFF's sin. Someone can easily shoot the leg is start two dataimport processes in parallel, you know. Solr doesn't really protects from 'lifecycle' errors, like accidentally enabled "autocommit"
- don't you think that Resolving the issue with such strick verdict can scare away other committers. Can you ping someone as well you expressed you point of view? I wanted to involve [~yseeley@gmail.com] or [~hossman_lucene@fucit.org], but hesitated to disturb them.
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Adrien Grand (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508114#comment-13508114 ]
Adrien Grand commented on SOLR-4085:
------------------------------------
Mikhail,
I took a quick look at your patch and I liked the fact that you didn't need to modify files outside of ExternalFileField and FileFLoatSouce to support commit-free updates. On the other hand, because of the lack of atomicity and the problem with concurrent reloads, I think it should not be applied to Solr because it could lead to confusing behaviors. (This doesn't mean that your work isn't valuable. As stated on http://wiki.apache.org/lucene-java/HowToContribute#Contributing_your_work, even patchs that are not committed are useful to the community.)
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-4085) Commit-free ExternalFileField
Posted by "Alan Woodward (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508789#comment-13508789 ]
Alan Woodward commented on SOLR-4085:
-------------------------------------
Hi Mikhail,
Sorry for the radio silence, I've been snowed under recently. I'll have a chance to have a proper look at this sometime this week.
I agree with Adrien that we should probably address the concurrent updates issue though; EFF is atomically updated at the moment, so this change could break existing installations.
> Commit-free ExternalFileField
> -----------------------------
>
> Key: SOLR-4085
> URL: https://issues.apache.org/jira/browse/SOLR-4085
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 4.1
> Reporter: Mikhail Khludnev
> Labels: externalfilefield
> Attachments: SOLR-4085.patch
>
>
> Let's reload ExternalFileFields without commit!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org