You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Jason Rutherglen (JIRA)" <ji...@apache.org> on 2008/05/07 03:13:55 UTC
[jira] Created: (SOLR-564) Realtime search in Solr
Realtime search in Solr
-----------------------
Key: SOLR-564
URL: https://issues.apache.org/jira/browse/SOLR-564
Project: Solr
Issue Type: New Feature
Components: replication, search
Affects Versions: 1.3
Reporter: Jason Rutherglen
Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
- Support for replication per update to transaction log
- Custom realtime index creation
- Filter and facet merging
- Custom IndexSearcher that ties into realtime subsystem
Is there a way to plug into these low level Solr functions without a massive redesign?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (SOLR-564) Realtime search in Solr
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
hi Jason ,
Have , the replication can probably piggyback on the new java
replication proposed (SOLR-561).
* It can be either a pull model . Keep the pollInterval very short (in
the slave) say every second or less and get the transaction logs. If
there is a new tx just get it and push it in the slave
* Push model. Register the urls of slaves in master . Trigger a
snappull (or a new txlogpull command ) in the slaves. The slaves will
then call the master to fetach the new logs and update it
--Noble
On Tue, Jun 3, 2008 at 1:11 AM, Jason Rutherglen (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Jason Rutherglen updated SOLR-564:
> ----------------------------------
>
> Attachment: solr-564.patch
>
> solr-564.patch
>
> Early Ocean Solr integration code. For viewing only. Depends on SOLR-568, SOLR-567, Lucene contrib InstantiatedIndex, LUCENE-1289, jdom, commons-io, commons-lang, geoapi-nogenerics-2.1-M2.jar, gt2-referencing-2.3.2.jar. Not sure what to do with heavily modified net.sourceforge.jsorter code (it is Apache license).
>
>> Realtime search in Solr
>> -----------------------
>>
>> Key: SOLR-564
>> URL: https://issues.apache.org/jira/browse/SOLR-564
>> Project: Solr
>> Issue Type: New Feature
>> Components: replication, search
>> Affects Versions: 1.3
>> Reporter: Jason Rutherglen
>> Attachments: solr-564.patch
>>
>>
>> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
>> - Support for replication per update to transaction log
>> - Custom realtime index creation
>> - Filter and facet merging
>> - Custom IndexSearcher that ties into realtime subsystem
>> - Custom SolrCore that ties into realtime subsystem
>> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
--Noble Paul
[jira] Closed: (SOLR-564) Realtime search in Solr
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen closed SOLR-564.
---------------------------------
Resolution: Invalid
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication (scripts), search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
> Attachments: solr-564.patch
>
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-564) Realtime search in Solr
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597635#action_12597635 ]
Hoss Man commented on SOLR-564:
-------------------------------
bq. Some issues that have been cropping up include the anywhere direct field cache access is performed. This includes SimpleFacets, ValueSource, and other places.
...like sorting.
bq. Performing work inside the segments and then merging the results later is how the Ocean is designed. I'm not sure how much work it would take to have Solr do the same.
this sounds akin to LUCENE-831 .. it should probably be dealt with at the Lucene level before attempting any big hoop jumping in Solr
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication, search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-564) Realtime search in Solr
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated SOLR-564:
----------------------------------
Description:
Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
- Support for replication per update to transaction log
- Custom realtime index creation
- Filter and facet merging
- Custom IndexSearcher that ties into realtime subsystem
- Custom SolrCore that ties into realtime subsystem
Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
was:
Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
- Support for replication per update to transaction log
- Custom realtime index creation
- Filter and facet merging
- Custom IndexSearcher that ties into realtime subsystem
Is there a way to plug into these low level Solr functions without a massive redesign?
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication, search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-564) Realtime search in Solr
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595837#action_12595837 ]
Hoss Man commented on SOLR-564:
-------------------------------
this seems like a "discussion" more then an "issue" ... as such it really belongs on solr-dev, not in Jira.
(if after some discussion, people come up with a plan to pursue, or a proposal that includes a patch, that's when a Jira issue starts to make sense)
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication, search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-564) Realtime search in Solr
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated SOLR-564:
----------------------------------
Attachment: solr-564.patch
solr-564.patch
Early Ocean Solr integration code. For viewing only. Depends on SOLR-568, SOLR-567, Lucene contrib InstantiatedIndex, LUCENE-1289, jdom, commons-io, commons-lang, geoapi-nogenerics-2.1-M2.jar, gt2-referencing-2.3.2.jar. Not sure what to do with heavily modified net.sourceforge.jsorter code (it is Apache license).
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication, search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
> Attachments: solr-564.patch
>
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-564) Realtime search in Solr
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595872#action_12595872 ]
Otis Gospodnetic commented on SOLR-564:
---------------------------------------
Jason, are your patches in various other Solr JIRA issues essentially the answer to your question above: Is there a way to plug into these low level Solr functions without a massive redesign? In other words, are you providing patches incrementally with the goal of getting Ocean Search's functionality merged into Solr? (that would be great!)
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication, search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-564) Realtime search in Solr
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595960#action_12595960 ]
Jason Rutherglen commented on SOLR-564:
---------------------------------------
The patches are a way to see if integration with Ocean is possible.
Some issues that have been cropping up include the anywhere direct field cache access is performed. This includes SimpleFacets, ValueSource, and other places. These are areas I do not have an immediate solution to and so the first version of OceanCore will not implement.
Because Ocean makes use field caching per segment, and new segments are being created all the time, something like MultiDocSet would need to be created to fit Ocean into the existing Lucene field cache framework. However this is impossible given Lucene field caching uses direct array access whereas DocSet is an interface. Ocean has a flexible field caching framework that includes a primitive based StringIndex like set of classes which save ram and allow fast range queries.
Performing work inside the segments and then merging the results later is how the Ocean is designed. I'm not sure how much work it would take to have Solr do the same. The concept is similar to the distributed indexing in Solr.
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication, search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-564) Realtime search in Solr
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598308#action_12598308 ]
Jason Rutherglen commented on SOLR-564:
---------------------------------------
After review, function queries ValueSource should be fine because the values are not loaded from the field cache until the query is executed in the sub-searcher. Sort is not mentioned because it is handled in the sub-searcher. LUCENE-831 is a good step, however, if used for the top level reader in a realtime system, large arrays will constantly be created for the top level reader after every transaction. This is why the work, meaning the query and the results, should be performed in the sub-searcher and then merged.
SimpleFacets.getFieldCacheCounts should be placed in the SolrIndexSearcher in order for it to be operable which will be placed in SOLR-567.
The Ocean Solr code patch will be attached to this issue.
> Realtime search in Solr
> -----------------------
>
> Key: SOLR-564
> URL: https://issues.apache.org/jira/browse/SOLR-564
> Project: Solr
> Issue Type: New Feature
> Components: replication, search
> Affects Versions: 1.3
> Reporter: Jason Rutherglen
>
> Before when I looked at this, the changes required to make Solr realtime would seem to break the rest of Solr. Is this still the case? In project Ocean http://code.google.com/p/oceansearch/ there is a realtime core however integrating into Solr has looked like a redesign of the guts of Solr.
> - Support for replication per update to transaction log
> - Custom realtime index creation
> - Filter and facet merging
> - Custom IndexSearcher that ties into realtime subsystem
> - Custom SolrCore that ties into realtime subsystem
> Is there a way to plug into these low level Solr functions without a massive redesign? A key area of concern is the doclist caching which is not used in realtime search because after every update the doclists are no longer valid. The doclist caching and handling is default in SolrCore. Ocean relies on a custom threaded MultiSearcher rather than a single IndexSearcher is a difficulty. DirectUpdateHandler2 works directly on IndexWriter is problematic.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.