You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/01/13 16:21:17 UTC
[GitHub] [solr] janhoy opened a new pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
janhoy opened a new pull request #525:
URL: https://github.com/apache/solr/pull/525
https://issues.apache.org/jira/browse/SOLR-15921
Makes it possible to add custom jars to a docker image as this
```dockerfile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #525:
URL: https://github.com/apache/solr/pull/525#issuecomment-1013762263
> The "problem" with the latter is only related to how we run Docker which oddly puts SOLR_HOME out in /var/solr which has always felt like a hack.
It has been a best practice for a long time to assign SOLR_HOME to your big data partition, and on `/var/xxx`, separate from the binary install which can be read-only. The `install_solr_service.sh` does the exact same thing. Oddly enough, the example configs do not respect $SOLR_HOME, so if you try `docker run --rm -p 8983:8983 solr -e techproducts` then it fails since it tries to write to `/opt/solr/example/techproducts/solr` which is and should be read-only.. It should have used `/var/solr/data/`.
So the issue of `SOLR_HOME/lib` not being co-located with the binary install is very old and was an issue long before Docker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
dsmiley commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r784993094
##########
File path: solr/solr-ref-guide/src/libs.adoc
##########
@@ -35,6 +35,10 @@ See <<taking-solr-to-production.adoc#solr-home-directory,Solr home directory>>.
* `<core_instance>/lib/`: In a user-managed cluster or a single-node installation, you may want to add plugins just for a specific Solr core.
Create this adjacent to the `conf/` directory; it's not present by default.
+* `<solr_install>/lib/`: The `.jar` files placed here are available to all Solr cores running on the node, and to node level plugins referenced in `solr.xml` -- so basically everything.
+Contrary to `<solr_home>/lib/`, this directory is always located in the install dir, so it can be used e.g. for custom
+Dockerfile to place custom plugin jars. This directory is not present by default so create it.
Review comment:
If we pre-create this then lets not overlook SOLR_HOME lib as well (not in Docker but in our distribution).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
dsmiley commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r784989503
##########
File path: solr/core/src/java/org/apache/solr/core/NodeConfig.java
##########
@@ -206,7 +208,16 @@ public Path getSolrDataHome() {
return solrDataHome;
}
- /**
+ /**
+ * Obtain the path of solr's binary installation directory, e.g. <code>/opt/solr</code>
+ * @return path to install dir, or null if property 'solr.install.dir' has not been initialized
+ */
+ public Path getSolrInstallDir() {
+ String prop = System.getProperty(SolrDispatchFilter.SOLR_INSTALL_DIR_ATTRIBUTE);
+ return prop != null ? Paths.get(prop) : null;
Review comment:
I like that you placed this here. Furthermore, perhaps SOLR_INSTALL_DIR_ATTRIBUTE actually belongs here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r785229738
##########
File path: solr/solr-ref-guide/src/libs.adoc
##########
@@ -35,6 +35,10 @@ See <<taking-solr-to-production.adoc#solr-home-directory,Solr home directory>>.
* `<core_instance>/lib/`: In a user-managed cluster or a single-node installation, you may want to add plugins just for a specific Solr core.
Create this adjacent to the `conf/` directory; it's not present by default.
+* `<solr_install>/lib/`: The `.jar` files placed here are available to all Solr cores running on the node, and to node level plugins referenced in `solr.xml` -- so basically everything.
+Contrary to `<solr_home>/lib/`, this directory is always located in the install dir, so it can be used e.g. for custom
+Dockerfile to place custom plugin jars. This directory is not present by default so create it.
Review comment:
I added the empty lib/ dir with a README in the latest commit.
Instead of adding yet another folder underneath `<solr-git>/solr/` with just a README, I put it in `solr/packaging/static`. Not very well thought through, can change that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #525:
URL: https://github.com/apache/solr/pull/525#issuecomment-1012561568
> Is there going to be an issue with the whole `allowPaths` thing? Or is anything under the solr-install-directory allowed?
I thought about it briefly. Since this path is coming from sys prop set by the start script it is not a user-entered value. And thus we would not need to validate it. I mean, this is THE source of thruth for what actually IS the `solrInstallDir`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r785208499
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -671,6 +671,10 @@ public void load() {
// Always add $SOLR_HOME/lib to the shared resource loader
Set<String> libDirs = new LinkedHashSet<>();
libDirs.add("lib");
+ // Always add $SOLR_TIP/lib to the shared resource loader, to allow loading of i.e. /opt/solr/lib/foo.jar
Review comment:
Ok, I did an attempt to move the sharedLib init (both SOLR_HOME/lib and SOLR_TIP/lib into NodeConfig. See latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
epugh commented on pull request #525:
URL: https://github.com/apache/solr/pull/525#issuecomment-1012298744
Looking forward to not having this in my Dockerfile!
```
COPY ./lib/streaming-expressions-1.0.0-solr8.5.0-SNAPSHOT.jar /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
HoustonPutman commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r784943516
##########
File path: solr/solr-ref-guide/src/libs.adoc
##########
@@ -35,6 +35,10 @@ See <<taking-solr-to-production.adoc#solr-home-directory,Solr home directory>>.
* `<core_instance>/lib/`: In a user-managed cluster or a single-node installation, you may want to add plugins just for a specific Solr core.
Create this adjacent to the `conf/` directory; it's not present by default.
+* `<solr_install>/lib/`: The `.jar` files placed here are available to all Solr cores running on the node, and to node level plugins referenced in `solr.xml` -- so basically everything.
+Contrary to `<solr_home>/lib/`, this directory is always located in the install dir, so it can be used e.g. for custom
+Dockerfile to place custom plugin jars. This directory is not present by default so create it.
Review comment:
I think that's likely a good idea, especially with a README, since it'd be strange to see an empty folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #525:
URL: https://github.com/apache/solr/pull/525#issuecomment-1015190990
> Note that there is another key place for documentation: libs.adoc.
This PR already adds information to `libs.adoc` :)
> That said, I really don't want to hold this PR forever
I'll target this for 9.1, and by then we'll know whether the efforts are overlapping or whether there is room for both.
Another thing I wanted to address is what kind of classloader the jars in `/lib` folder is added to. I have experienced before that certain types of plugins will not work in `$SOLR_HOME/lib`, you have to place them in `WEB-INF/lib`. That is super confusing. I think it is Jetty trying to isolate webapps from each other, but Solr is only one app so we should need no such extra isolation. I don't remember what kind of plugins this was, perhaps it was those with SPI loading or using reflection? Any way I think we should try to let the new`lib/` location be equivalent to `WEB-INF/lib` so it can accept any kind of plugins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r785208882
##########
File path: solr/core/src/java/org/apache/solr/core/NodeConfig.java
##########
@@ -157,6 +162,8 @@ private NodeConfig(String nodeName, Path coreRootDirectory, Path solrDataHome, I
}
if (null == this.solrHome) throw new NullPointerException("solrHome");
if (null == this.loader) throw new NullPointerException("loader");
+
+ setupSharedLib();
Review comment:
I found no better place to kick this off than in the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
dsmiley commented on pull request #525:
URL: https://github.com/apache/solr/pull/525#issuecomment-1013283366
I like how easily this solves a problem but it creates yet another dir that seems very duplicitous with SOLR_HOME lib. The "problem" with the latter is only related to how we run Docker which oddly puts SOLR_HOME out in /var/solr which has always felt like a hack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r784928616
##########
File path: solr/core/src/java/org/apache/solr/core/NodeConfig.java
##########
@@ -206,7 +208,16 @@ public Path getSolrDataHome() {
return solrDataHome;
}
- /**
+ /**
+ * Obtain the path of solr's binary installation directory, e.g. <code>/opt/solr</code>
+ * @return path to install dir, or null if property 'solr.install.dir' has not been initialized
+ */
+ public Path getSolrInstallDir() {
+ String prop = System.getProperty(SolrDispatchFilter.SOLR_INSTALL_DIR_ATTRIBUTE);
+ return prop != null ? Paths.get(prop) : null;
Review comment:
I was a little weary about returning null here. Alternatives would be String which could be "" or `Optional<Path>` but since solr.install.dir is always set, both in a normal run and in test, that is a corner case I did not bother with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r784930720
##########
File path: solr/core/src/java/org/apache/solr/core/NodeConfig.java
##########
@@ -206,7 +208,16 @@ public Path getSolrDataHome() {
return solrDataHome;
}
- /**
+ /**
+ * Obtain the path of solr's binary installation directory, e.g. <code>/opt/solr</code>
+ * @return path to install dir, or null if property 'solr.install.dir' has not been initialized
+ */
+ public Path getSolrInstallDir() {
+ String prop = System.getProperty(SolrDispatchFilter.SOLR_INSTALL_DIR_ATTRIBUTE);
+ return prop != null ? Paths.get(prop) : null;
Review comment:
I also thought of not adding this method to `NodeConfig` and instead use `System.getProperty` directly where you need it - like is done other places in the code base. But it IS a node configuration and fits nicely in there. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r785211424
##########
File path: solr/core/src/java/org/apache/solr/core/NodeConfig.java
##########
@@ -206,7 +208,16 @@ public Path getSolrDataHome() {
return solrDataHome;
}
- /**
+ /**
+ * Obtain the path of solr's binary installation directory, e.g. <code>/opt/solr</code>
+ * @return path to install dir, or null if property 'solr.install.dir' has not been initialized
+ */
+ public Path getSolrInstallDir() {
+ String prop = System.getProperty(SolrDispatchFilter.SOLR_INSTALL_DIR_ATTRIBUTE);
+ return prop != null ? Paths.get(prop) : null;
Review comment:
DispatchFilter has six different sysprop constants defined already. I think perhaps instead of spreading them all around we should factor out a new class `SolrSystemProperties` to keep all sysprop contants in one place? But that is a refactoring for another day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #525:
URL: https://github.com/apache/solr/pull/525#issuecomment-1014916175
I think this is ready.
Do you see a conflict between this and your future `$SOLR_VAR` / `$SOLR_HOME` refactoring, @dsmiley ? If you want me to hold this for a week or to until you get further along with that effort, we could perhaps view them together and consolidate in some way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
dsmiley commented on pull request #525:
URL: https://github.com/apache/solr/pull/525#issuecomment-1015043179
Note that there is another key place for documentation: `libs.adoc`. This is where it will be most apparent that this is yet another place that is equivalent to `solr_home/lib`.
Thanks for considering the future refactoring... yeah please lets wait. That said, I really don't want to hold this PR forever if that doesn't come to fruition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r784931851
##########
File path: solr/solr-ref-guide/src/libs.adoc
##########
@@ -35,6 +35,10 @@ See <<taking-solr-to-production.adoc#solr-home-directory,Solr home directory>>.
* `<core_instance>/lib/`: In a user-managed cluster or a single-node installation, you may want to add plugins just for a specific Solr core.
Create this adjacent to the `conf/` directory; it's not present by default.
+* `<solr_install>/lib/`: The `.jar` files placed here are available to all Solr cores running on the node, and to node level plugins referenced in `solr.xml` -- so basically everything.
+Contrary to `<solr_home>/lib/`, this directory is always located in the install dir, so it can be used e.g. for custom
+Dockerfile to place custom plugin jars. This directory is not present by default so create it.
Review comment:
Would it be better if the tarball had an empty `lib/` folder, or possibly with a README? Then you'd not have to do a `mkdir` before `cp` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a change in pull request #525: SOLR-15921 Look for jars in SOLR_TIP/lib/
Posted by GitBox <gi...@apache.org>.
dsmiley commented on a change in pull request #525:
URL: https://github.com/apache/solr/pull/525#discussion_r784987859
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -671,6 +671,10 @@ public void load() {
// Always add $SOLR_HOME/lib to the shared resource loader
Set<String> libDirs = new LinkedHashSet<>();
libDirs.add("lib");
+ // Always add $SOLR_TIP/lib to the shared resource loader, to allow loading of i.e. /opt/solr/lib/foo.jar
Review comment:
I'm further reminded of my earlier code review in which I asked that such stuff be the job of NodeConfig and not CoreContainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org