You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Michael Gibney <mi...@michaelgibney.net> on 2023/09/07 16:13:57 UTC
updateLog tlog directory config silently ignored (SOLR-16962)
I'd like to call some extra attention to SOLR-16962 [1]. I'm fairly
certain that it's been years since it has actually worked to specify
tlog dir in solrconfig.xml `<updateLog/>` element. I've opened a PR
that fixes this, but it involves making some decisions about edge-case
handling, for which I'd appreciate feedback.
Of course, if anyone _has_ successfully configured a custom tlog dir
location with a recent solr version, I'd be curious to hear about that
too!
Also worth mentioning: if anyone currently has this latently
configured in solrconfig.xml, I wonder if it's possible that fixing
this bug could cause a change in behavior that might cause a
backcompat issue.
[1] https://issues.apache.org/jira/browse/SOLR-16962
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: updateLog tlog directory config silently ignored (SOLR-16962)
Posted by Michael Gibney <mi...@michaelgibney.net>.
Any further thoughts on this?
I'd like to proceed with the PR for SOLR-16962 [1], which I've updated
with a fix to address now-once-again-relevant SOLR-6537 [2] (at
whatever point SOLR-16962 made it impossible to configure an external
tlog, SOLR-6537 (cleanup/deletion of external tlog) became practically
not relevant).
[1] https://issues.apache.org/jira/browse/SOLR-16962
[2] https://issues.apache.org/jira/browse/SOLR-6537
On Mon, Sep 11, 2023 at 12:36 PM Michael Gibney
<mi...@michaelgibney.net> wrote:
>
> > useful to separate the location to use some fast local SSDs vs slower but larger storage for the main index
>
> Exactly.
>
> And yes, the duplicity of nomenclature is a large part of what I'm
> suggesting to deal with here ... it's very confusing. But having
> recently been pretty deep in this part of the current code, it looks
> very much like "ulogDir" refers to the _parent_ directory of the tlog
> dir -- i.e., `[instanceDir]/data/`, which is just weird IMO ... so I
> guess I was also thinking of consistency, but hoping to break more in
> the direction of using "tlogDir", since "ulogDir" currently
> (confusingly) seems to mean `[instanceDir]/data/`.
>
> My current understanding of the situation: as far as the filesystem is
> concerned, updateLog/ulog/solr.ulog.dir/etc. has no significance other
> than as the parent of the tlogDir -- and as the parent, it also
> happens to be the same dir as the default dataDir -- i.e., the parent
> of the `index` dir, etc... so in fact there's nothing at all uniquely
> "ulog" about the "ulogDir".
>
> Michael
>
> On Mon, Sep 11, 2023 at 12:24 PM David Smiley <da...@gmail.com> wrote:
> >
> > FWIW I've never heard of anyone customizing that. I suppose it might be
> > useful to separate the location to use some fast local SSDs vs slower but
> > larger storage for the main index?
> >
> > I've noticed an annoying duplicity in nomenclature here -- is it the Update
> > Log or the Transaction Log? And furthermore, we have a TLOG replica type.
> > In our configuration, the nomenclature is "update log" or "ulog". If we
> > add a new setting, let's at least be consistent and call the new proposed
> > dir "ulogDir".
> >
> > ~ David
> >
> >
> > On Mon, Sep 11, 2023 at 9:45 AM Michael Gibney <mi...@michaelgibney.net>
> > wrote:
> >
> > > Thanks Ilan, this seems good. In addition to addressing practical
> > > concerns, this could also be an opportunity to clarify the semantics
> > > around updateLog/tlog configuration:
> > >
> > > In looking closely at this code, my understanding is that the
> > > `<updateLog><str name="dir">${solr.ulog.dir:}</str></updateLog>`
> > > setting in solrconfig.xml currently defines the _parent_ directory
> > > that contains the tlog dir. Except that afaict, the tlog dir is the
> > > _only_ path that's actually relevant. In fact, the portion of the
> > > refguide that mentions this is called "Transaction Log Configuration"
> > > [1].
> > >
> > > This muddles the intent of this setting; it would be much clearer if,
> > > rather than a "dir" property (defaulting to `[instanceDir]/data/`)
> > > defining the _parent_ directory of the tlog dir, we simply had a
> > > "tlogDir" property (defaulting to `[instanceDir]/data/tlog/`) that
> > > directly defines the tlog dir. The semantics of this could then be
> > > clarified to indicate that:
> > > 1. if tlogDir is relative, it is resolved relative to core
> > > instanceDir, and must resolve to a descendent dir of instanceDir
> > > (i.e., cannot "escape" via ../../../).
> > > 2. if tlogDir (whether absolute or relative) resolves to a descendant
> > > of instanceDir, it is already implicitly namespaced by core, and the
> > > property defines the tlog dir directly.
> > > 3. if tlogDir is absolute and resolves to a directory _outside_ of the
> > > core instanceDir, it is _not_ implicitly namespaced by core, and the
> > > property therefore defines a parent directory, within which a
> > > core-namespaced tlog dir will be created (for a final tlog file path
> > > of, e.g., `[tlogDir]/[core_name]/tlog.0000000000000001`)
> > >
> > > As it stands currently, the above three cases are already ambiguous;
> > > the proposed bugfix PR [2] necessarily resolves this ambiguity, but
> > > even so, the semantics of the "dir" property map poorly onto what
> > > actually needs to be configured.
> > >
> > > Does this make sense, and/or am I missing something?
> > >
> > > Michael
> > >
> > > [1]
> > > https://solr.apache.org/guide/solr/latest/configuration-guide/commits-transaction-logs.html#transaction-log-configuration
> > > [2] https://github.com/apache/solr/pull/1895
> > >
> > > On Fri, Sep 8, 2023 at 4:59 PM Ilan Ginzburg <il...@gmail.com> wrote:
> > > >
> > > > Re backcompat issue: might be an option to fix that config but use a
> > > > different name for it then, so that any currently existing (and ignored)
> > > > config continues to be ignored.
> > > >
> > > > Ilan
> > > >
> > > > On Thu, Sep 7, 2023, 6:14 PM Michael Gibney <mi...@michaelgibney.net>
> > > > wrote:
> > > >
> > > > > I'd like to call some extra attention to SOLR-16962 [1]. I'm fairly
> > > > > certain that it's been years since it has actually worked to specify
> > > > > tlog dir in solrconfig.xml `<updateLog/>` element. I've opened a PR
> > > > > that fixes this, but it involves making some decisions about edge-case
> > > > > handling, for which I'd appreciate feedback.
> > > > >
> > > > > Of course, if anyone _has_ successfully configured a custom tlog dir
> > > > > location with a recent solr version, I'd be curious to hear about that
> > > > > too!
> > > > >
> > > > > Also worth mentioning: if anyone currently has this latently
> > > > > configured in solrconfig.xml, I wonder if it's possible that fixing
> > > > > this bug could cause a change in behavior that might cause a
> > > > > backcompat issue.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/SOLR-16962
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: updateLog tlog directory config silently ignored (SOLR-16962)
Posted by Michael Gibney <mi...@michaelgibney.net>.
> useful to separate the location to use some fast local SSDs vs slower but larger storage for the main index
Exactly.
And yes, the duplicity of nomenclature is a large part of what I'm
suggesting to deal with here ... it's very confusing. But having
recently been pretty deep in this part of the current code, it looks
very much like "ulogDir" refers to the _parent_ directory of the tlog
dir -- i.e., `[instanceDir]/data/`, which is just weird IMO ... so I
guess I was also thinking of consistency, but hoping to break more in
the direction of using "tlogDir", since "ulogDir" currently
(confusingly) seems to mean `[instanceDir]/data/`.
My current understanding of the situation: as far as the filesystem is
concerned, updateLog/ulog/solr.ulog.dir/etc. has no significance other
than as the parent of the tlogDir -- and as the parent, it also
happens to be the same dir as the default dataDir -- i.e., the parent
of the `index` dir, etc... so in fact there's nothing at all uniquely
"ulog" about the "ulogDir".
Michael
On Mon, Sep 11, 2023 at 12:24 PM David Smiley <da...@gmail.com> wrote:
>
> FWIW I've never heard of anyone customizing that. I suppose it might be
> useful to separate the location to use some fast local SSDs vs slower but
> larger storage for the main index?
>
> I've noticed an annoying duplicity in nomenclature here -- is it the Update
> Log or the Transaction Log? And furthermore, we have a TLOG replica type.
> In our configuration, the nomenclature is "update log" or "ulog". If we
> add a new setting, let's at least be consistent and call the new proposed
> dir "ulogDir".
>
> ~ David
>
>
> On Mon, Sep 11, 2023 at 9:45 AM Michael Gibney <mi...@michaelgibney.net>
> wrote:
>
> > Thanks Ilan, this seems good. In addition to addressing practical
> > concerns, this could also be an opportunity to clarify the semantics
> > around updateLog/tlog configuration:
> >
> > In looking closely at this code, my understanding is that the
> > `<updateLog><str name="dir">${solr.ulog.dir:}</str></updateLog>`
> > setting in solrconfig.xml currently defines the _parent_ directory
> > that contains the tlog dir. Except that afaict, the tlog dir is the
> > _only_ path that's actually relevant. In fact, the portion of the
> > refguide that mentions this is called "Transaction Log Configuration"
> > [1].
> >
> > This muddles the intent of this setting; it would be much clearer if,
> > rather than a "dir" property (defaulting to `[instanceDir]/data/`)
> > defining the _parent_ directory of the tlog dir, we simply had a
> > "tlogDir" property (defaulting to `[instanceDir]/data/tlog/`) that
> > directly defines the tlog dir. The semantics of this could then be
> > clarified to indicate that:
> > 1. if tlogDir is relative, it is resolved relative to core
> > instanceDir, and must resolve to a descendent dir of instanceDir
> > (i.e., cannot "escape" via ../../../).
> > 2. if tlogDir (whether absolute or relative) resolves to a descendant
> > of instanceDir, it is already implicitly namespaced by core, and the
> > property defines the tlog dir directly.
> > 3. if tlogDir is absolute and resolves to a directory _outside_ of the
> > core instanceDir, it is _not_ implicitly namespaced by core, and the
> > property therefore defines a parent directory, within which a
> > core-namespaced tlog dir will be created (for a final tlog file path
> > of, e.g., `[tlogDir]/[core_name]/tlog.0000000000000001`)
> >
> > As it stands currently, the above three cases are already ambiguous;
> > the proposed bugfix PR [2] necessarily resolves this ambiguity, but
> > even so, the semantics of the "dir" property map poorly onto what
> > actually needs to be configured.
> >
> > Does this make sense, and/or am I missing something?
> >
> > Michael
> >
> > [1]
> > https://solr.apache.org/guide/solr/latest/configuration-guide/commits-transaction-logs.html#transaction-log-configuration
> > [2] https://github.com/apache/solr/pull/1895
> >
> > On Fri, Sep 8, 2023 at 4:59 PM Ilan Ginzburg <il...@gmail.com> wrote:
> > >
> > > Re backcompat issue: might be an option to fix that config but use a
> > > different name for it then, so that any currently existing (and ignored)
> > > config continues to be ignored.
> > >
> > > Ilan
> > >
> > > On Thu, Sep 7, 2023, 6:14 PM Michael Gibney <mi...@michaelgibney.net>
> > > wrote:
> > >
> > > > I'd like to call some extra attention to SOLR-16962 [1]. I'm fairly
> > > > certain that it's been years since it has actually worked to specify
> > > > tlog dir in solrconfig.xml `<updateLog/>` element. I've opened a PR
> > > > that fixes this, but it involves making some decisions about edge-case
> > > > handling, for which I'd appreciate feedback.
> > > >
> > > > Of course, if anyone _has_ successfully configured a custom tlog dir
> > > > location with a recent solr version, I'd be curious to hear about that
> > > > too!
> > > >
> > > > Also worth mentioning: if anyone currently has this latently
> > > > configured in solrconfig.xml, I wonder if it's possible that fixing
> > > > this bug could cause a change in behavior that might cause a
> > > > backcompat issue.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/SOLR-16962
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: updateLog tlog directory config silently ignored (SOLR-16962)
Posted by David Smiley <da...@gmail.com>.
FWIW I've never heard of anyone customizing that. I suppose it might be
useful to separate the location to use some fast local SSDs vs slower but
larger storage for the main index?
I've noticed an annoying duplicity in nomenclature here -- is it the Update
Log or the Transaction Log? And furthermore, we have a TLOG replica type.
In our configuration, the nomenclature is "update log" or "ulog". If we
add a new setting, let's at least be consistent and call the new proposed
dir "ulogDir".
~ David
On Mon, Sep 11, 2023 at 9:45 AM Michael Gibney <mi...@michaelgibney.net>
wrote:
> Thanks Ilan, this seems good. In addition to addressing practical
> concerns, this could also be an opportunity to clarify the semantics
> around updateLog/tlog configuration:
>
> In looking closely at this code, my understanding is that the
> `<updateLog><str name="dir">${solr.ulog.dir:}</str></updateLog>`
> setting in solrconfig.xml currently defines the _parent_ directory
> that contains the tlog dir. Except that afaict, the tlog dir is the
> _only_ path that's actually relevant. In fact, the portion of the
> refguide that mentions this is called "Transaction Log Configuration"
> [1].
>
> This muddles the intent of this setting; it would be much clearer if,
> rather than a "dir" property (defaulting to `[instanceDir]/data/`)
> defining the _parent_ directory of the tlog dir, we simply had a
> "tlogDir" property (defaulting to `[instanceDir]/data/tlog/`) that
> directly defines the tlog dir. The semantics of this could then be
> clarified to indicate that:
> 1. if tlogDir is relative, it is resolved relative to core
> instanceDir, and must resolve to a descendent dir of instanceDir
> (i.e., cannot "escape" via ../../../).
> 2. if tlogDir (whether absolute or relative) resolves to a descendant
> of instanceDir, it is already implicitly namespaced by core, and the
> property defines the tlog dir directly.
> 3. if tlogDir is absolute and resolves to a directory _outside_ of the
> core instanceDir, it is _not_ implicitly namespaced by core, and the
> property therefore defines a parent directory, within which a
> core-namespaced tlog dir will be created (for a final tlog file path
> of, e.g., `[tlogDir]/[core_name]/tlog.0000000000000001`)
>
> As it stands currently, the above three cases are already ambiguous;
> the proposed bugfix PR [2] necessarily resolves this ambiguity, but
> even so, the semantics of the "dir" property map poorly onto what
> actually needs to be configured.
>
> Does this make sense, and/or am I missing something?
>
> Michael
>
> [1]
> https://solr.apache.org/guide/solr/latest/configuration-guide/commits-transaction-logs.html#transaction-log-configuration
> [2] https://github.com/apache/solr/pull/1895
>
> On Fri, Sep 8, 2023 at 4:59 PM Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > Re backcompat issue: might be an option to fix that config but use a
> > different name for it then, so that any currently existing (and ignored)
> > config continues to be ignored.
> >
> > Ilan
> >
> > On Thu, Sep 7, 2023, 6:14 PM Michael Gibney <mi...@michaelgibney.net>
> > wrote:
> >
> > > I'd like to call some extra attention to SOLR-16962 [1]. I'm fairly
> > > certain that it's been years since it has actually worked to specify
> > > tlog dir in solrconfig.xml `<updateLog/>` element. I've opened a PR
> > > that fixes this, but it involves making some decisions about edge-case
> > > handling, for which I'd appreciate feedback.
> > >
> > > Of course, if anyone _has_ successfully configured a custom tlog dir
> > > location with a recent solr version, I'd be curious to hear about that
> > > too!
> > >
> > > Also worth mentioning: if anyone currently has this latently
> > > configured in solrconfig.xml, I wonder if it's possible that fixing
> > > this bug could cause a change in behavior that might cause a
> > > backcompat issue.
> > >
> > > [1] https://issues.apache.org/jira/browse/SOLR-16962
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: updateLog tlog directory config silently ignored (SOLR-16962)
Posted by Michael Gibney <mi...@michaelgibney.net>.
Thanks Ilan, this seems good. In addition to addressing practical
concerns, this could also be an opportunity to clarify the semantics
around updateLog/tlog configuration:
In looking closely at this code, my understanding is that the
`<updateLog><str name="dir">${solr.ulog.dir:}</str></updateLog>`
setting in solrconfig.xml currently defines the _parent_ directory
that contains the tlog dir. Except that afaict, the tlog dir is the
_only_ path that's actually relevant. In fact, the portion of the
refguide that mentions this is called "Transaction Log Configuration"
[1].
This muddles the intent of this setting; it would be much clearer if,
rather than a "dir" property (defaulting to `[instanceDir]/data/`)
defining the _parent_ directory of the tlog dir, we simply had a
"tlogDir" property (defaulting to `[instanceDir]/data/tlog/`) that
directly defines the tlog dir. The semantics of this could then be
clarified to indicate that:
1. if tlogDir is relative, it is resolved relative to core
instanceDir, and must resolve to a descendent dir of instanceDir
(i.e., cannot "escape" via ../../../).
2. if tlogDir (whether absolute or relative) resolves to a descendant
of instanceDir, it is already implicitly namespaced by core, and the
property defines the tlog dir directly.
3. if tlogDir is absolute and resolves to a directory _outside_ of the
core instanceDir, it is _not_ implicitly namespaced by core, and the
property therefore defines a parent directory, within which a
core-namespaced tlog dir will be created (for a final tlog file path
of, e.g., `[tlogDir]/[core_name]/tlog.0000000000000001`)
As it stands currently, the above three cases are already ambiguous;
the proposed bugfix PR [2] necessarily resolves this ambiguity, but
even so, the semantics of the "dir" property map poorly onto what
actually needs to be configured.
Does this make sense, and/or am I missing something?
Michael
[1] https://solr.apache.org/guide/solr/latest/configuration-guide/commits-transaction-logs.html#transaction-log-configuration
[2] https://github.com/apache/solr/pull/1895
On Fri, Sep 8, 2023 at 4:59 PM Ilan Ginzburg <il...@gmail.com> wrote:
>
> Re backcompat issue: might be an option to fix that config but use a
> different name for it then, so that any currently existing (and ignored)
> config continues to be ignored.
>
> Ilan
>
> On Thu, Sep 7, 2023, 6:14 PM Michael Gibney <mi...@michaelgibney.net>
> wrote:
>
> > I'd like to call some extra attention to SOLR-16962 [1]. I'm fairly
> > certain that it's been years since it has actually worked to specify
> > tlog dir in solrconfig.xml `<updateLog/>` element. I've opened a PR
> > that fixes this, but it involves making some decisions about edge-case
> > handling, for which I'd appreciate feedback.
> >
> > Of course, if anyone _has_ successfully configured a custom tlog dir
> > location with a recent solr version, I'd be curious to hear about that
> > too!
> >
> > Also worth mentioning: if anyone currently has this latently
> > configured in solrconfig.xml, I wonder if it's possible that fixing
> > this bug could cause a change in behavior that might cause a
> > backcompat issue.
> >
> > [1] https://issues.apache.org/jira/browse/SOLR-16962
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: updateLog tlog directory config silently ignored (SOLR-16962)
Posted by Ilan Ginzburg <il...@gmail.com>.
Re backcompat issue: might be an option to fix that config but use a
different name for it then, so that any currently existing (and ignored)
config continues to be ignored.
Ilan
On Thu, Sep 7, 2023, 6:14 PM Michael Gibney <mi...@michaelgibney.net>
wrote:
> I'd like to call some extra attention to SOLR-16962 [1]. I'm fairly
> certain that it's been years since it has actually worked to specify
> tlog dir in solrconfig.xml `<updateLog/>` element. I've opened a PR
> that fixes this, but it involves making some decisions about edge-case
> handling, for which I'd appreciate feedback.
>
> Of course, if anyone _has_ successfully configured a custom tlog dir
> location with a recent solr version, I'd be curious to hear about that
> too!
>
> Also worth mentioning: if anyone currently has this latently
> configured in solrconfig.xml, I wonder if it's possible that fixing
> this bug could cause a change in behavior that might cause a
> backcompat issue.
>
> [1] https://issues.apache.org/jira/browse/SOLR-16962
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>