Re: poor performance when connecting to CloudSolrServer(zkHosts) using solrJ

[Mohmed Hussain <mo...@gmail.com>]

My two cents, do check network connectivity. In past I remember changing
the zookeeper server name to actual IP improved the speed a bit.
DNS sometimes take time to resolve hostname. Could be worth trying this
option.


Thanks
-Hussain

On Mon, Dec 29, 2014 at 6:31 PM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 12/29/2014 6:52 PM, zhangjianad@dcits.com wrote:
> >       I setups a SolrCloud, and code a simple solrJ program to query solr
> > data as below, but it takes about 40 seconds to new CloudSolrServer
> > instance,less than 100 miliseconds is acceptable. what is going on when
> new
> > CloudSolrServer? and how to fix this issue?
> >
> >       String zkHost = "bicenter1.dcc:2181,datanode2.dcc:2181";
> >       String defaultCollection = "hdfsCollection";
> >
> >       long startms=System.currentTimeMillis();
> >       CloudSolrServer server = new CloudSolrServer(zkHost);
> >       server.setDefaultCollection(defaultCollection);
> >       server.setZkConnectTimeout(3000);
> >       server.setZkClientTimeout(6000);
> >       long endms=System.currentTimeMillis();
> >       System.out.println(endms-startms);
> >
> >       ModifiableSolrParams params = new ModifiableSolrParams();
> >       params.set("q", "id:*hbase*");
> >       params.set("sort", "price desc");
> >       params.set("start", "0");
> >       params.set("rows", "10");
> >
> >       try {
> >               QueryResponse response=server.query(params);
> >               SolrDocumentList results = response.getResults();
> >               for (SolrDocument doc:results) {
> >                       String rowkey=doc.getFieldValue("id").toString();
> >               }
> >
> >       } catch (SolrServerException e) {
> >               // TODO Auto-generated catch block
> >               e.printStackTrace();
> >       }
> >
> >       server.shutdown();
>
> The only part of the constructor for CloudSolrServer that I cannot
> easily look at is the part that creates the httpclient, because
> ultimately that calls code outside of Solr, in the HttpComponents
> project.  Everything that I *can* see is code that should happen
> extremely quickly, and the httpclient creation code is something that I
> have used myself and never had any noticeable delay.  The constructor
> for CloudSolrServer does *NOT* contact zookeeper or Solr, it merely sets
> up the instance.  Nothing is contacted until a request is made.  I
> examined the CloudSolrServer code from branch_5x.
>
> I tried out your code (with SolrJ 4.6.0 against a SolrCloud 4.2.1
> cluster).  Although the query itself encountered an exception in
> zookeeper (probably from the version discrepancy between Solr and
> SolrJ), the elapsed time printed out from the CloudSolrServer
> initialization was 240 milliseconds on the first run, 60 milliseconds on
> a second run, and 64 milliseconds on a third run.  Those are all MUCH
> less than the 1000 milliseconds that would represent one second, and
> incredibly less than the 40000 milliseconds that would represent 40
> seconds.
>
> Side issue:  I hope that you have more than two zookeeper servers in
> your ensemble.  A two-node zookeeper ensemble is actually *less*
> reliable than a single node, because a failure of EITHER of those two
> nodes will result in a loss of quorum.  Three nodes is the minimum
> required for a redundant zookeeper ensemble.
>
> Thanks,
> Shawn
>
>

RE: poor performance when connecting to CloudSolrServer(zkHosts) using solrJ

[steve <sc...@hotmail.com>]

While I'm not a net optimization whiz, a properly configured DNS client will "cache" the recent resolved lookups; this way even though you are referring to the Fully Qualified Domain Name (FQDN), the local DNS client will return the recently acquired IP address (within the constraints of the Domain's configuration). In other words, while there is "overhead" between the local workstation/computer and the DNS client, it will NOT require access to the configured DNS server "upstream".
Enjoy,Steve

> Date: Thu, 1 Jan 2015 14:30:19 -0800
> Subject: Re: poor performance when connecting to CloudSolrServer(zkHosts) using solrJ
> From: mohd.hussain@gmail.com
> To: solr-user@lucene.apache.org
> 
> My two cents, do check network connectivity. In past I remember changing
> the zookeeper server name to actual IP improved the speed a bit.
> DNS sometimes take time to resolve hostname. Could be worth trying this
> option.
> 
> 
> Thanks
> -Hussain
> 
> On Mon, Dec 29, 2014 at 6:31 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> 
> > On 12/29/2014 6:52 PM, zhangjianad@dcits.com wrote:
> > >       I setups a SolrCloud, and code a simple solrJ program to query solr
> > > data as below, but it takes about 40 seconds to new CloudSolrServer
> > > instance,less than 100 miliseconds is acceptable. what is going on when
> > new
> > > CloudSolrServer? and how to fix this issue?
> > >
> > >       String zkHost = "bicenter1.dcc:2181,datanode2.dcc:2181";
> > >       String defaultCollection = "hdfsCollection";
> > >
> > >       long startms=System.currentTimeMillis();
> > >       CloudSolrServer server = new CloudSolrServer(zkHost);
> > >       server.setDefaultCollection(defaultCollection);
> > >       server.setZkConnectTimeout(3000);
> > >       server.setZkClientTimeout(6000);
> > >       long endms=System.currentTimeMillis();
> > >       System.out.println(endms-startms);
> > >
> > >       ModifiableSolrParams params = new ModifiableSolrParams();
> > >       params.set("q", "id:*hbase*");
> > >       params.set("sort", "price desc");
> > >       params.set("start", "0");
> > >       params.set("rows", "10");
> > >
> > >       try {
> > >               QueryResponse response=server.query(params);
> > >               SolrDocumentList results = response.getResults();
> > >               for (SolrDocument doc:results) {
> > >                       String rowkey=doc.getFieldValue("id").toString();
> > >               }
> > >
> > >       } catch (SolrServerException e) {
> > >               // TODO Auto-generated catch block
> > >               e.printStackTrace();
> > >       }
> > >
> > >       server.shutdown();
> >
> > The only part of the constructor for CloudSolrServer that I cannot
> > easily look at is the part that creates the httpclient, because
> > ultimately that calls code outside of Solr, in the HttpComponents
> > project.  Everything that I *can* see is code that should happen
> > extremely quickly, and the httpclient creation code is something that I
> > have used myself and never had any noticeable delay.  The constructor
> > for CloudSolrServer does *NOT* contact zookeeper or Solr, it merely sets
> > up the instance.  Nothing is contacted until a request is made.  I
> > examined the CloudSolrServer code from branch_5x.
> >
> > I tried out your code (with SolrJ 4.6.0 against a SolrCloud 4.2.1
> > cluster).  Although the query itself encountered an exception in
> > zookeeper (probably from the version discrepancy between Solr and
> > SolrJ), the elapsed time printed out from the CloudSolrServer
> > initialization was 240 milliseconds on the first run, 60 milliseconds on
> > a second run, and 64 milliseconds on a third run.  Those are all MUCH
> > less than the 1000 milliseconds that would represent one second, and
> > incredibly less than the 40000 milliseconds that would represent 40
> > seconds.
> >
> > Side issue:  I hope that you have more than two zookeeper servers in
> > your ensemble.  A two-node zookeeper ensemble is actually *less*
> > reliable than a single node, because a failure of EITHER of those two
> > nodes will result in a loss of quorum.  Three nodes is the minimum
> > required for a redundant zookeeper ensemble.
> >
> > Thanks,
> > Shawn
> >
> >