Data migration from one collection to the other collection

[vidya <vi...@tcs.com>]

Hi

I would like to maintain two cores for history data and current data where
hdfs is my datasource. My requirement is that data input should be given to
only one collection and previous data should be moved to history collection.
1)Creating two cores and migrating data from current to history collection
by data-config.xml using solrEntityProcessor. In data-config.xml, where
should i represent two collections for migrating source collection to the
other collection.And how to make sure that happens.Do I need to run a job or
how to make sure that data migration occurs.
https://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor

2)collection aliasing is a concept which creates new collection after a
period of time.
I read that concept but lagging in how to implement it.Like where do i need
to make changes in my solrcloud.
http://blog.cloudera.com/blog/2013/10/collection-aliasing-near-real-time-search-for-really-big-data/

Please help me on this.

Thanks in advance



--
View this message in context: http://lucene.472066.n3.nabble.com/Data-migration-from-one-collection-to-the-other-collection-tp4248646.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data migration from one collection to the other collection

[Walter Underwood <wu...@wunderwood.org>]

You could send the documents to both and filter out the recent ones in the history collection.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jan 5, 2016, at 5:46 AM, vidya <vi...@tcs.com> wrote:
> 
> Hi
> 
> I would like to maintain two cores for history data and current data where
> hdfs is my datasource. My requirement is that data input should be given to
> only one collection and previous data should be moved to history collection.
> 1)Creating two cores and migrating data from current to history collection
> by data-config.xml using solrEntityProcessor. In data-config.xml, where
> should i represent two collections for migrating source collection to the
> other collection.And how to make sure that happens.Do I need to run a job or
> how to make sure that data migration occurs.
> https://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor
> 
> 2)collection aliasing is a concept which creates new collection after a
> period of time.
> I read that concept but lagging in how to implement it.Like where do i need
> to make changes in my solrcloud.
> http://blog.cloudera.com/blog/2013/10/collection-aliasing-near-real-time-search-for-really-big-data/
> 
> Please help me on this.
> 
> Thanks in advance
> 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Data-migration-from-one-collection-to-the-other-collection-tp4248646.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data migration from one collection to the other collection

[Erick Erickson <er...@gmail.com>]

What changes? You simply have "hot" and "cold" collections. When it comes time
to index data you:
1> create a collection
2> index to it.
3> use the Collections API to point your "active" collection to this new one
4> do whatever you want with the old one.

The setup is, of course, that your hot and cold collections are never
pointed to by the url. So your first time around you index to, say,
collection1. Then you create an alias to it with the collections API. Call
the alias "searchcollection".

At this point search collection points to collection1.
Now you index to collection2
Now you use the collections API point searchcolleciton to collection1

The url for search always uses "searchcollection"

Best,
Erick


On Tue, Jan 5, 2016 at 5:46 AM, vidya <vi...@tcs.com> wrote:
> Hi
>
> I would like to maintain two cores for history data and current data where
> hdfs is my datasource. My requirement is that data input should be given to
> only one collection and previous data should be moved to history collection.
> 1)Creating two cores and migrating data from current to history collection
> by data-config.xml using solrEntityProcessor. In data-config.xml, where
> should i represent two collections for migrating source collection to the
> other collection.And how to make sure that happens.Do I need to run a job or
> how to make sure that data migration occurs.
> https://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor
>
> 2)collection aliasing is a concept which creates new collection after a
> period of time.
> I read that concept but lagging in how to implement it.Like where do i need
> to make changes in my solrcloud.
> http://blog.cloudera.com/blog/2013/10/collection-aliasing-near-real-time-search-for-really-big-data/
>
> Please help me on this.
>
> Thanks in advance
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Data-migration-from-one-collection-to-the-other-collection-tp4248646.html
> Sent from the Solr - User mailing list archive at Nabble.com.