Configuring alias in Solr 6.5

[Phil Budne <ph...@ultimate.com>]

Hello Solr World,

I'm a firefighter, brought in to help nurse a Solr Cloud system back
to life until it can be replaced.  It's running Solr 6.5, and a data
item (key) field exceeded what will fit in a signed 32-bit collection
field a while ago.

I've been able to make a new collection with 64-bit fields, and new
data is being inserted, and an alias created using:
http://shard-01:8983/solr/admin/collections?action=CREATEALIAS&name=alias&collections=collection2,collection1

Works as hoped/expected for both queries and inserts.

BUT, the alias definition does not persist between restarts.  I
haven't spotted any newly written (aliases.json?) file(s) created on
any of the shards, or the zookeper system (all of the shards and
zookeeper are running in Docker containers).

It's entirely possible this is due to permission issues (but no errors
appeared in the log file on the shard where I issued the command).

This is for a non-profit/academic research organization with a tight
budget (and I'm helping them recover from an extended period of things
not working at all), so "doing it right", upgrading etc may not be in
the cards, and I'm hoping I just need to create an aliases.json file
of the right format, in the right location.

I tried creating an aliases.json file on all of the systems, in the
location suggested by, and with the same data shown by zookeeper cli
"list":

 /aliases.json (0)
 DATA:
     {"collection":{"alias":"collection2,collection1"}}
 /collections (2)
  /collections/collection1 (2)
  ....

The new collection didn't work until I issued zkcli upconfig &
linkconfig commands that are issued when the zookeeper container
starts, so perhaps I just need some additional zkcli incantations?

Thanks In Advance!
Phil Budne

Re: Configuring alias in Solr 6.5

[Shawn Heisey <ap...@elyograg.org>]

On 3/26/2022 3:42 PM, Phil Budne wrote:
> How do I configure the node hostname?
> I see "-Dhost=${HOSTNAME}" being passed to the jvm on startup.

The actual setting of that value is accomplished in solr.xml.  The 
default stock solr.xml file has these three lines under the <solrcloud> 
section.  These use system properties, set using the syntax you showed:

     <str name="host">${host:}</str>
     <int name="hostPort">${solr.port.advertise:0}</int>
     <str name="hostContext">${hostContext:solr}</str>

The ":0" on the second line there seems wrong to me, but I haven't heard 
about any big problems with node registration in ZK, so it must be OK.

Thanks,
Shawn

Re: Configuring alias in Solr 6.5

[Phil Budne <ph...@ultimate.com>]

Shawn wrote:
> As a workaround to the "problem" they are trying to solve ... each Solr 
> node can be configured to send a different string for its hostname to 
> zookeeper than its actual hostname. Then you just make sure DNS is 
> always up to date and uses a low TTL value for the docker-related names.

That sounds worth a try.  I'm seeing it take a while to come on line as-is.

How do I configure the node hostname?
I see "-Dhost=${HOSTNAME}" being passed to the jvm on startup.

Docker is already providing a resolvable name based on the docker
service name (and each shard is a service), so I shouldn't have to do
anything there, EXCEPT that the reverse (PTR) record will still be the
dynamic name of the container.

Re: Configuring alias in Solr 6.5

[Shawn Heisey <ap...@elyograg.org>]

On 3/25/2022 9:49 PM, Phil Budne wrote:
> Thanks for the pictures.  I think I may have it figured out.
>
> The Dockerfile for zookeeper runs upconf and linkconf for each
> collection, and then saves a copy of /var/lib/zookeeper duing
> construction of the image, and doesn't persist any changes made while
> running.  I found this comment:
>
>      # Move the "clean" ZooKeeper data directory to /var/lib/zookeeper-template/
>      # so that we could use a clean data directory (e.g. without Solr hostnames
>      # hardcoded) on every ZooKeeper start.

So when there is a restart they basically wipe the ZK database to a 
"clean" state?  That seems like fixing an improperly placed nail for 
hanging a painting by bulldozing and rebuilding the entire house.  Sure, 
it will work, but it's silly.

> So can I preload the aliases data, perhaps with the zkcli "putfile" command?

That should work.  Or maybe you could figure out how to build that into 
the ZK template.

As a workaround to the "problem" they are trying to solve ... each Solr 
node can be configured to send a different string for its hostname to 
zookeeper than its actual hostname.  Then you just make sure DNS is 
always up to date and uses a low TTL value for the docker-related names.

Thanks,
Shawn

Re: Configuring alias in Solr 6.5

[Phil Budne <ph...@ultimate.com>]

Thanks for the pictures.  I think I may have it figured out.

The Dockerfile for zookeeper runs upconf and linkconf for each
collection, and then saves a copy of /var/lib/zookeeper duing
construction of the image, and doesn't persist any changes made while
running.  I found this comment:

    # Move the "clean" ZooKeeper data directory to /var/lib/zookeeper-template/
    # so that we could use a clean data directory (e.g. without Solr hostnames
    # hardcoded) on every ZooKeeper start.

Perhaps because they were having trouble with dynamic container
instance hostnames ending up being persisted. From zkcli "list":

>                  "base_url":"http://a6ca9ef1259c:8983/solr",
>                  "node_name":"a6ca9ef1259c:8983_solr",

So can I preload the aliases data, perhaps with the zkcli "putfile" command?

Thanks!
Phil

Re: Configuring alias in Solr 6.5

[Shawn Heisey <ap...@elyograg.org>]

On 3/25/2022 5:04 PM, Phil Budne wrote:
>> All that should be in zookeeper, not on the filesystem.
> Where should zookeeper stash the data?

The aliases.json file is at the root of the zookeeper tree.

https://www.dropbox.com/s/fc5clxz211ydesh/Screenshot%202022-03-25%2017.40.17.png?dl=0

I tried 6.6.6 and got the same behavior as the 6.5.x versions.

https://www.dropbox.com/s/d1lv8gjghnkkhm8/Screenshot%202022-03-25%2017.40.55.png?dl=0

https://www.dropbox.com/s/0edc9g6aojrm677/Screenshot%202022-03-25%2017.41.02.png?dl=0

Thanks,
Shawn

Re: Configuring alias in Solr 6.5

[Phil Budne <ph...@ultimate.com>]

Shawn,

Thanks for your reply!

> All that should be in zookeeper, not on the filesystem.

Where should zookeeper stash the data?

> Don't go by what the admin UI shows you. Try accessing 
> http://server:port/solr/xxxx/select?q=*:* ... substituting the correct 
> server, port, and alias name.

Both inserts and queries against the alias failed after restart of the
shards and zookeeper.

As I wrote, I'm doing life support, and don't expect any fixes.
Right now the program inserting data checks if Solr is up by issuing a
query that should always find no entries.  If worst comes to worst,
I'll have it try to create the alias if the query returns a 404.

Phil

Re: Configuring alias in Solr 6.5

[Shawn Heisey <ap...@elyograg.org>]

On 3/25/2022 12:04 PM, Phil Budne wrote:
> BUT, the alias definition does not persist between restarts.  I
> haven't spotted any newly written (aliases.json?) file(s) created on
> any of the shards, or the zookeper system (all of the shards and
> zookeeper are running in Docker containers).

All that should be in zookeeper, not on the filesystem.  If you're 
looking for aliases.json on disk, you won't find it unless you create 
it.  And any file like that which you create will not be used.

I have never heard of aliases disappearing when a SolrCloud node 
restarts.  And if it does disappear somehow, then it should affect ALL 
the Solr nodes in the cluster, not just the one that restarted.  I did 
some searches in Jira, the bug tracker, and didn't find anything.

I just tried the cloud example on 8.11.1 and it functioned exactly as 
expected.  I created an alias, tested it, restarted both nodes, and the 
alias was still there.

Solr 6.5.0 and 6.5.1 seem to be a little different.  It appears that the 
admin UI in this version does NOT display aliases.  But the alias worked 
after I created it, and still worked after restarting both nodes.  But 
the alias NEVER showed in the admin UI, either before or after the 
restarts.  So Solr appears to be working correctly, but the admin UI may 
not show aliases.  I did the alias creation in the admin UI, so I found 
it very strange that it didn't show there.

There will be no more 6.x releases.  Soon there won't even be more 8.x 
releases except for major show-stopper or security bugs ... the project 
is in the process of releasing 9.0.0.

Don't go by what the admin UI shows you.  Try accessing 
http://server:port/solr/xxxx/select?q=*:* ... substituting the correct 
server, port, and alias name.

Thanks,
Shawn