You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Brendan Grainger <br...@motologic.com> on 2016/01/15 20:51:17 UTC
state.json base_url has internal IP of ec2 instance set instead of 'public DNS' entry in some cases
Hi,
I am creating a new collection using the following get request:
http://ec2_host:8983/solr/admin/collections?action=CREATE&name=collection_name_1&collection.configName=oem/conf&numShards=1
What I’m finding is that now and then base_url for the replica in state.json is set to the internal IP of the AWS node. i.e.:
"base_url":"http://10.29.XXX.XX:8983/solr”,
On other attempts it’s set to the public DNS name of the node:
"base_url":"http://ec2_host:8983/solr”,
In my /etc/defaults/solr.in.sh I have:
SOLR_HOST=“ec2_host”
which I thought is what I needed to get the public DNS name set in base_url.
Am I doing this incorrectly or is there something I’m missing here? The issue this causes is zookeeper gives back an internal IP to my indexing processes when the internal IP is set on base_url and they then can’t find the server.
Thanks!
Re: state.json base_url has internal IP of ec2 instance set instead of 'public DNS' entry in some cases
Posted by Brendan Grainger <br...@motologic.com>.
Hi Hoss,
Thanks for your help. Going over the install page again I realized I had originally not adjusted the value of SOLR_HOST and it had started up using the default internal IP. I changed that to the public DNS and restarted solr. However in /live_nodes I then had 2 values: one for the public DNS and one for the internal IP. It looks like it didn’t get removed. I removed it using the zookeeper cli and all is working fine now.
I’m unsure, but wondering if the behavior I saw is somehow related to this: http://www.gossamer-threads.com/lists/lucene/java-dev/297790 <http://www.gossamer-threads.com/lists/lucene/java-dev/297790> However, as I said I’m pretty new to this so I could be completely wrong.
Thanks again
Brendan
> On Jan 15, 2016, at 6:07 PM, Chris Hostetter <ho...@fucit.org> wrote:
>
>
> : What I’m finding is that now and then base_url for the replica in
> : state.json is set to the internal IP of the AWS node. i.e.:
> :
> : "base_url":"http://10.29.XXX.XX:8983/solr”,
> :
> : On other attempts it’s set to the public DNS name of the node:
> :
> : "base_url":"http://ec2_host:8983/solr”,
> :
> : In my /etc/defaults/solr.in.sh I have:
> :
> : SOLR_HOST=“ec2_host”
> :
> : which I thought is what I needed to get the public DNS name set in base_url.
>
> i believe you are correct. the "now and then" part of your question is
> weird -- it seems to indicate that sometimes the "correct" thing is
> happening, and other times it is not.
>
> /etc/defaults/solr.in.sh isn't the canonical path for solr.in.sh
> according to the docs/install script for running a production solr
> instance...
>
> https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production#TakingSolrtoProduction-ServiceInstallationScript
>
> ...how *exactly* are you running Solr on all of your nodes?
>
> because my guess is that you've got some kind of inconsistent setup where
> sometimes when you startup (or restart) a new node it does refer to your
> solr.in.sh file, and other times it does not -- so sometimes solr never
> sees your SOLR_HOST option. In those cases, when it regesters itself with
> ZooKeeper it uses the current IP as a fallback, and then that info gets
> backed into the metadata for the replicas that get created on that node
> at that point in time.
>
> FWIW, you should be able to spot check that the SOLR_HOST is being applied
> correctly by looking at the java process command line args (using PS, or
> loading the SOlr UI in your browser) and checking for the "-Dhost=..."
> option -- if it's not there, then your solr.in.sh probably wasn't read in
> correctly
>
>
>
> -Hoss
> http://www.lucidworks.com/
Re: state.json base_url has internal IP of ec2 instance set instead of 'public DNS' entry in some cases
Posted by Brendan Grainger <br...@motologic.com>.
Hi Hoss,
Thanks for the reply. I installed the service using the install script. I double checked it and it looks like it install solr.in.sh in /etc/defaults/solr.in.sh. It actually looks like if it is in /var the install script moves it into /etc/defaults (unless I’m reading this wrong):
https://github.com/apache/lucene-solr/blob/trunk/solr/bin/install_solr_service.sh#L281 <https://github.com/apache/lucene-solr/blob/trunk/solr/bin/install_solr_service.sh#L281>
I checked the process and even on restarts it looks like this:
ps aux | grep solr
my_solr_user 9522 0.2 1.5 3010216 272656 ? Sl 20:06 0:26 /usr/lib/jvm/java-8-oracle/bin/java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/solr/logs/solr_gc.log -DzkClientTimeout=15000 -DzkHost=ec2_host:2181/solr -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Dhost=ec2_host -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/var/solr/data -Dsolr.install.dir=/opt/solr -Dlog4j.configuration=file:/var/solr/log4j.properties -Xss256k -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs --module=http
Note I replaced the user I’m running it as with my_solr_user and the actual ec2 public DNS with e2_host in the output above.
I am new to SolrCloud so it’s more than likely I’ve screwed up some configuration setting somewhere.
Thank you for your help,
Brendan
> On Jan 15, 2016, at 6:07 PM, Chris Hostetter <ho...@fucit.org> wrote:
>
>
> : What I’m finding is that now and then base_url for the replica in
> : state.json is set to the internal IP of the AWS node. i.e.:
> :
> : "base_url":"http://10.29.XXX.XX:8983/solr”,
> :
> : On other attempts it’s set to the public DNS name of the node:
> :
> : "base_url":"http://ec2_host:8983/solr”,
> :
> : In my /etc/defaults/solr.in.sh I have:
> :
> : SOLR_HOST=“ec2_host”
> :
> : which I thought is what I needed to get the public DNS name set in base_url.
>
> i believe you are correct. the "now and then" part of your question is
> weird -- it seems to indicate that sometimes the "correct" thing is
> happening, and other times it is not.
>
> /etc/defaults/solr.in.sh isn't the canonical path for solr.in.sh
> according to the docs/install script for running a production solr
> instance...
>
> https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production#TakingSolrtoProduction-ServiceInstallationScript
>
> ...how *exactly* are you running Solr on all of your nodes?
>
> because my guess is that you've got some kind of inconsistent setup where
> sometimes when you startup (or restart) a new node it does refer to your
> solr.in.sh file, and other times it does not -- so sometimes solr never
> sees your SOLR_HOST option. In those cases, when it regesters itself with
> ZooKeeper it uses the current IP as a fallback, and then that info gets
> backed into the metadata for the replicas that get created on that node
> at that point in time.
>
> FWIW, you should be able to spot check that the SOLR_HOST is being applied
> correctly by looking at the java process command line args (using PS, or
> loading the SOlr UI in your browser) and checking for the "-Dhost=..."
> option -- if it's not there, then your solr.in.sh probably wasn't read in
> correctly
>
>
>
> -Hoss
> http://www.lucidworks.com/
Re: state.json base_url has internal IP of ec2 instance set instead
of 'public DNS' entry in some cases
Posted by Chris Hostetter <ho...@fucit.org>.
: What I’m finding is that now and then base_url for the replica in
: state.json is set to the internal IP of the AWS node. i.e.:
:
: "base_url":"http://10.29.XXX.XX:8983/solr”,
:
: On other attempts it’s set to the public DNS name of the node:
:
: "base_url":"http://ec2_host:8983/solr”,
:
: In my /etc/defaults/solr.in.sh I have:
:
: SOLR_HOST=“ec2_host”
:
: which I thought is what I needed to get the public DNS name set in base_url.
i believe you are correct. the "now and then" part of your question is
weird -- it seems to indicate that sometimes the "correct" thing is
happening, and other times it is not.
/etc/defaults/solr.in.sh isn't the canonical path for solr.in.sh
according to the docs/install script for running a production solr
instance...
https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production#TakingSolrtoProduction-ServiceInstallationScript
...how *exactly* are you running Solr on all of your nodes?
because my guess is that you've got some kind of inconsistent setup where
sometimes when you startup (or restart) a new node it does refer to your
solr.in.sh file, and other times it does not -- so sometimes solr never
sees your SOLR_HOST option. In those cases, when it regesters itself with
ZooKeeper it uses the current IP as a fallback, and then that info gets
backed into the metadata for the replicas that get created on that node
at that point in time.
FWIW, you should be able to spot check that the SOLR_HOST is being applied
correctly by looking at the java process command line args (using PS, or
loading the SOlr UI in your browser) and checking for the "-Dhost=..."
option -- if it's not there, then your solr.in.sh probably wasn't read in
correctly
-Hoss
http://www.lucidworks.com/