Re: hot shard concept

[hupadhyay <hu...@asite.com>]

Shawn this seems to be a great implementation work.

 I am also trying to do something similar. But i am planning to use
SolrCloud which manages shards and replication automatically,pls make me
correct if i am wrong. I had explored mongoDB and cassandra too and mongoDB
was fair enough to be used, but the main concern was creating and
maintaining the index.

Same issue still persist with SolrCloud, but dataimport handler come to
rescue.

My data base is too large around 2-3TB and is ACL based.

I am able to create initial full index with 4 shard configuration with 10
machines cluster using dataimport handler. But the real isseu starts now,
how to maintain index up to date along with database, cause my search needs
to be a near real time. There will be around few millions of updates in a
day and the index needs to be up to date.

As a result of this i have planned to implement JMS interface for async
update in to index along with db.
This seems to be a very heavy task considering above mentioned update counts
per day.

The root cause for this heavy load is per document level ACL implementation
in solr, say each solr doc ships a list of users who have access to it,which
incurs a heavy load of update on index.

Can you pls suggest any alternate for this to maintain index up to date,
along with needs your input on how to manage ACL ?

Thanks a lot.



--
View this message in context: http://lucene.472066.n3.nabble.com/hot-shard-concept-tp4016676p4017491.html
Sent from the Solr - User mailing list archive at Nabble.com.