Storing application metadata for a collection? / APIs to read properties and configuration

[Colvin Cowie <co...@gmail.com>]

Hello,

I'm revisiting something that I did a long time ago, and looking to tidy
things up but I think I need to add at least one bit of API to do it
(unless there's a different way to go).

I've got an application which indexes to Solr, and it records some metadata
for each collection - e.g. the last database row that was indexed, flags to
indicate whether indexing has been paused, things like that - trivially
represented as a JSON file. For some specific collections I also need to
store a XML file which contains some configuration.

For several reasons (we wanted those files to be included in backup+restore
operations from Solr, and we wanted the metadata to be wiped if a
collection was deleted and recreated) we decided that it made sense to
store the files in ZooKeeper within the nodes used by Solr, so we ended up
storing the metadata properties in the collectionprops.json, and the XML
inside the configset used by the collection (with the collection name used
to uniquely name it within a shared config). The application uses a
ZooKeeper client to directly interact with those files in ZooKeeper.

That worked well enough so far, but it would definitely be cleaner and
safer to go through Solr Admin APIs instead so that the application doesn't
need to talk to ZooKeeper or know about Solr's internals.

For the properties my first thought was that I could just use the
COLLECTIONPROP
API
<https://solr.apache.org/guide/solr/latest/deployment-guide/collection-management.html#collectionprop>
as I'm already storing them in collectionprops.json. However, it doesn't
have a read operation, it only has set and delete operations... Would
adding a GET to the v2 API be reasonable?
I see that https://issues.apache.org/jira/browse/SOLR-12224 did suggest
that would happen as part of
https://issues.apache.org/jira/browse/SOLR-15734, but it doesn't look like
there's an active issue for it. I'd be happy to have a look at implementing
it in a few weeks if it is something that's generally wanted.

And for the XML file, I could use the Configset API
<https://solr.apache.org/guide/solr/latest/configuration-guide/configsets-api.html#configsets-upload>
to upload it, but again there's no API to download the file (nor even the
configset as a whole). Would adding APIs to fetch the content of a
configset be generally useful (or objectionable)?

Though configsets aren't the best fit for this anyway, the file really
belongs to an individual collection's configuration rather than a shareable
configset. Instead with a read API for collection properties, I could
base64 encode my XML and store it as a collection property instead. The
downside to that would be that it makes the collectionprops.json bigger
than it needs to be and lead to a lot of churn in ZooKeeper. The XML is
written once while the metadata properties are written many times, and I
don't want Solr to care about it either... it doesn't need to be watched
etc.

So in many ways, what I really would like is to have a generic API to store
and retrieve "stuff" for a collection (which is already achievable using
the ZK client), and have it included in the backup+restore of a collection
(which doesn't happen currently AFAIK, only specific nodes are restored).
i.e. a CRUD API that just delegates straight to the ZooKeeper client and
stores things under the collection (under a well known node).
Perhaps that is too niche and nobody else would need it? Or maybe I should
solve the whole problem in a different way in the first place?

Regards,
Colvin

Re: Storing application metadata for a collection? / APIs to read properties and configuration

[David Smiley <ds...@apache.org>]

Yes exactly -- would be really cool!

For example, imagine if the V2 API for this was
here: /api/c/collection-name/config/files and you could HTTP PUT
config/files/synonyms.txt
And furthermore GET of files or listings would present the overlaid view.

This is all wishful thinking of course; it's some serious work to totally
complete the vision.  If you (or your colleagues, etc.) are interested in
pursuing this, I could help define the work.

On Wed, Jan 8, 2025 at 9:12 AM Colvin Cowie <co...@gmail.com>
wrote:

>  I'm not entirely familiar with that "auto-created" configSet, but I think
> I see what you mean. If your configuration is all within a collection
> specific overlay you wouldn't even need to delete a config set, you would
> just delete the collection (or possibly the overlay), so would entirely
> avoid the loop in
>
> https://github.com/apache/solr/blob/388101fc84140c0bdc50706637bcc5df11908ca6/solr/core/src/java/org/apache/solr/cloud/ConfigSetCmds.java#L186-L197
>
> I can see an overlay being a useful general solution that adds value within
> Solr and externally too
>
>    - stored under the collection
>       - included in backup and restore
>       - deleted when you delete the collection
>       - allows you to make collection specific modifications to override
>    common config - within reason - while still receiving updates to the
> common
>    parts.
>       - changing the entire content of a znode would be okay
>       - "hiding/deleting" a znode (or its children) from the common config
>       would be more complicated / harder to reason about, and probably
> isn't
>       generally necessary?
>       - an API that looks something like the existing configset API (with
>    read operations)
>
>
> On Tue, 7 Jan 2025 at 17:20, David Smiley <ds...@apache.org> wrote:
>
> > If we were to pursue my proposal, it'd mean that the auto-created
> configSet
> > thing (creating a mutable configSet from an immutable template) could go
> > away, which is a source of some complexity and scale concerns (loops all
> > collections when certain configSets get deleted).
> >
> > On Tue, Jan 7, 2025 at 10:33 AM David Smiley <ds...@apache.org> wrote:
> >
> > > Where I work, we've used collection properties (collectionprops.json)
> to
> > > store collection specific synonyms & query-elevation configuration.  We
> > > have custom plugins for those features to read/expire this information
> in
> > > addition to implementing other custom requirements.  It's somewhat of a
> > > hack but it works while being rather lean implementation-wise as it
> uses
> > an
> > > existing mechanism.
> > >
> > > It'd be interesting if a collection could have a configset overlay as a
> > > general feature and for modifying anything.  It'd be stored inside a
> > > collection's ZK node tree.  SolrResourceLoader would overlay this with
> > the
> > > backing (ConfigSet) source.
> > >
> >
>

Re: Storing application metadata for a collection? / APIs to read properties and configuration

[Colvin Cowie <co...@gmail.com>]

 I'm not entirely familiar with that "auto-created" configSet, but I think
I see what you mean. If your configuration is all within a collection
specific overlay you wouldn't even need to delete a config set, you would
just delete the collection (or possibly the overlay), so would entirely
avoid the loop in
https://github.com/apache/solr/blob/388101fc84140c0bdc50706637bcc5df11908ca6/solr/core/src/java/org/apache/solr/cloud/ConfigSetCmds.java#L186-L197

I can see an overlay being a useful general solution that adds value within
Solr and externally too

   - stored under the collection
      - included in backup and restore
      - deleted when you delete the collection
      - allows you to make collection specific modifications to override
   common config - within reason - while still receiving updates to the common
   parts.
      - changing the entire content of a znode would be okay
      - "hiding/deleting" a znode (or its children) from the common config
      would be more complicated / harder to reason about, and probably isn't
      generally necessary?
      - an API that looks something like the existing configset API (with
   read operations)


On Tue, 7 Jan 2025 at 17:20, David Smiley <ds...@apache.org> wrote:

> If we were to pursue my proposal, it'd mean that the auto-created configSet
> thing (creating a mutable configSet from an immutable template) could go
> away, which is a source of some complexity and scale concerns (loops all
> collections when certain configSets get deleted).
>
> On Tue, Jan 7, 2025 at 10:33 AM David Smiley <ds...@apache.org> wrote:
>
> > Where I work, we've used collection properties (collectionprops.json) to
> > store collection specific synonyms & query-elevation configuration.  We
> > have custom plugins for those features to read/expire this information in
> > addition to implementing other custom requirements.  It's somewhat of a
> > hack but it works while being rather lean implementation-wise as it uses
> an
> > existing mechanism.
> >
> > It'd be interesting if a collection could have a configset overlay as a
> > general feature and for modifying anything.  It'd be stored inside a
> > collection's ZK node tree.  SolrResourceLoader would overlay this with
> the
> > backing (ConfigSet) source.
> >
>

Re: Storing application metadata for a collection? / APIs to read properties and configuration

[David Smiley <ds...@apache.org>]

If we were to pursue my proposal, it'd mean that the auto-created configSet
thing (creating a mutable configSet from an immutable template) could go
away, which is a source of some complexity and scale concerns (loops all
collections when certain configSets get deleted).

On Tue, Jan 7, 2025 at 10:33 AM David Smiley <ds...@apache.org> wrote:

> Where I work, we've used collection properties (collectionprops.json) to
> store collection specific synonyms & query-elevation configuration.  We
> have custom plugins for those features to read/expire this information in
> addition to implementing other custom requirements.  It's somewhat of a
> hack but it works while being rather lean implementation-wise as it uses an
> existing mechanism.
>
> It'd be interesting if a collection could have a configset overlay as a
> general feature and for modifying anything.  It'd be stored inside a
> collection's ZK node tree.  SolrResourceLoader would overlay this with the
> backing (ConfigSet) source.
>

Re: Storing application metadata for a collection? / APIs to read properties and configuration

[David Smiley <ds...@apache.org>]

Where I work, we've used collection properties (collectionprops.json) to
store collection specific synonyms & query-elevation configuration.  We
have custom plugins for those features to read/expire this information in
addition to implementing other custom requirements.  It's somewhat of a
hack but it works while being rather lean implementation-wise as it uses an
existing mechanism.

It'd be interesting if a collection could have a configset overlay as a
general feature and for modifying anything.  It'd be stored inside a
collection's ZK node tree.  SolrResourceLoader would overlay this with the
backing (ConfigSet) source.