[jira] Issue Comment Edited: (SOLR-1277) Implement a Solr specific naming service (using Zookeeper)

["Mark Miller (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784317#action_12784317 ] 

Mark Miller edited comment on SOLR-1277 at 12/1/09 6:18 PM:
------------------------------------------------------------

I'd like to reboot this issue a bit. I've started a fresh experimental pass.

I think we want to move ZooKeeper's home out of SolrCore and into CoreContainer - one of the big benefits of using ZooKeeper is the ability to config from a central location. This implies that a ZooKeeper client should be up and running before any SolrCore is.

So this new patch explores that path - its just an initial stab in that direction, so a lot is left undone, but I figured I'd put it up to get feedback on the new approach.

This initial pass allows basic loading of config/schema from ZooKeeper and adds some ground work for ZooKeeper unit tests.

Some simple notes below:
{noformat}
ZooKeeper Integration

Features:

load solrconfig/schema from zookeeper (currently no other config files - want to be able to load any config from ZK)

TODO: handle distributed search node choices, fault tolerance
TODO: integrate with replication
TODO: fault tolerant indexing
TODO: more ;)

Startup?

System Properties? -DzkHost=localhost:2127 {-DzkPath=/solr} - if no zkHost found, run non ZooKeeper mode

Optional Properties? -Dsolrconfig=solrconfig.xml -Dschema=schema.xml - multicore config: comes from solr.xml

Groups? -Dgroup= ? -Dshard= ? How about multicore? Set in solr.xml?

Distinguish master from slave?
 Another sys property?

Cache zk info locally and 'watch' for changes


Layout - configurable prefix - defaults to solr

/solr/conf
         /solrconfig.xml
         /schema.xml

multicore?

/solr/core0/conf
/solr/core1/conf


ZooKeeper Limitations:

1 MB of data per node
wants zk log on its own device (not just partition)
 (perhaps not so necessary with fewer nodes?)
must not swap to maintain performance
{noformat}

      was (Author: markrmiller@gmail.com):
    I'd like to reboot this issue a bit. I've started a fresh experimental pass.

I think we want to move ZooKeeper's home out of SolrCore and into CoreContainer - one of the big benefits of using ZooKeeper is the ability to config from a central location. This implies that a ZooKeeper client should be up and running before any SolrCore is.

So this new patch explores that path - its just an initial stab in that direction, so a lot is left on done, but I figured I'd put it up to get feedback on the new approach.

This initial pass allows basic loading of config/schema from ZooKeeper and adds some ground work for ZooKeeper unit tests.

Some simple notes below:
{noformat}
ZooKeeper Integration

Features:

load solrconfig/schema from zookeeper (currently no other config files - want to be able to load any config from ZK)

TODO: handle distributed search node choices, fault tolerance
TODO: integrate with replication
TODO: fault tolerant indexing
TODO: more ;)

Startup?

System Properties? -DzkHost=localhost:2127 {-DzkPath=/solr} - if no zkHost found, run non ZooKeeper mode

Optional Properties? -Dsolrconfig=solrconfig.xml -Dschema=schema.xml - multicore config: comes from solr.xml

Groups? -Dgroup= ? -Dshard= ? How about multicore? Set in solr.xml?

Distinguish master from slave?
 Another sys property?

Cache zk info locally and 'watch' for changes


Layout - configurable prefix - defaults to solr

/solr/conf
         /solrconfig.xml
         /schema.xml

multicore?

/solr/core0/conf
/solr/core1/conf


ZooKeeper Limitations:

1 MB of data per node
wants zk log on its own device (not just partition)
 (perhaps not so necessary with fewer nodes?)
must not swap to maintain performance
{noformat}
  
> Implement a Solr specific naming service (using Zookeeper)
> ----------------------------------------------------------
>
>                 Key: SOLR-1277
>                 URL: https://issues.apache.org/jira/browse/SOLR-1277
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: log4j-1.2.15.jar, SOLR-1277.patch, SOLR-1277.patch, zookeeper-3.2.1.jar
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> The goal is to give Solr server clusters self-healing attributes
> where if a server fails, indexing and searching don't stop and
> all of the partitions remain searchable. For configuration, the
> ability to centrally deploy a new configuration without servers
> going offline.
> We can start with basic failover and start from there?
> Features:
> * Automatic failover (i.e. when a server fails, clients stop
> trying to index to or search it)
> * Centralized configuration management (i.e. new solrconfig.xml
> or schema.xml propagates to a live Solr cluster)
> * Optionally allow shards of a partition to be moved to another
> server (i.e. if a server gets hot, move the hot segments out to
> cooler servers). Ideally we'd have a way to detect hot segments
> and move them seamlessly. With NRT this becomes somewhat more
> difficult but not impossible?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.