You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "HoustonPutman (via GitHub)" <gi...@apache.org> on 2023/07/07 16:21:10 UTC
[GitHub] [solr] HoustonPutman opened a new pull request, #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
HoustonPutman opened a new pull request, #1765:
URL: https://github.com/apache/solr/pull/1765
https://issues.apache.org/jira/browse/SOLR-16878
This can be tested relatively easily, **using Java 20**
```
$ ./gradlew dev
$ cd solr/packaging/build/dev
$ ./bin/solr start -e films
```
In the solr logs (not the console logs), you will see:
```
INFO (coreLoadExecutor-10-thread-1) [ x:films] o.a.l.s.MemorySegmentIndexInputProvider Using MemorySegmentIndexInput with Java 20; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
```
For the new Vector logs, after introducing the option in Solr in #1758, no logging is done by default. But we can use the C1 compiler, which makes Lucene log that it can't use the Panama Vector API.
```
$ ./gradlew dev
$ cd solr/packaging/build/dev
$ ./bin/solr start -e films -a "-XX:TieredStopAtLevel=1"
```
In the solr logs (again, not the console logs), you will see:
```
WARN (qtp1027495011-41) [ x:films] o.a.l.u.VectorUtilProvider C2 compiler is disabled; Java vector incubator API can't be enabled
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1629499919
Oh it is merged already! Mhm, maybe remove the obsolete line, otherwise all is fine!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on a diff in pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #1765:
URL: https://github.com/apache/solr/pull/1765#discussion_r1258682017
##########
solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java:
##########
@@ -226,6 +229,21 @@ public void init(ServletContext servletContext) {
StartupLoggingUtils.changeLogLevel(logLevel);
}
+ // Do initial logs for experimental Lucene classes.
+ // TODO: Use "MethodHandles.lookup().ensureClassInitialized()" instead of "Class.forName()"
+ // once JDK 15+ is mandatory
+ Stream.of(MMapDirectory.class, VectorUtil.class)
+ .forEach(
+ cls -> {
+ try {
+ Class.forName(cls.getName());
+ } catch (ReflectiveOperationException re) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "Could not load Lucene class: " + cls.getName());
+ }
+ });
+ VectorUtil.dotProduct(new byte[0], new byte[0]);
Review Comment:
If the class is initialized (what the previous code does), this call is obsolete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1625705293
It should also log if it is enabled: https://github.com/apache/lucene/blob/d03c8f16d9d4f87ce24d603e4a9b4a7806f9b533/lucene/core/src/java20/org/apache/lucene/internal/vectorization/PanamaVectorizationProvider.java#L74
Maybe the issue is that VectorUtil is not used on IndexReader or IndexWriter init.
Tip: Elasticsearch triggers class initialization using `MethodHandles.lookup().ensureClassInitialited(VectorUtil.class)`. Same for MMapDirectory. This enables warnings and similar directly on startup.
I am in cinema, so will later look into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1765:
URL: https://github.com/apache/solr/pull/1765#discussion_r1258309737
##########
solr/bin/solr:
##########
@@ -1581,7 +1581,8 @@ if [ "${SOLR_HOME:0:${#EXAMPLE_DIR}}" = "$EXAMPLE_DIR" ]; then
SOLR_LOGS_DIR="$SOLR_HOME/../logs"
fi
-LOG4J_CONFIG=()
+# Set the logging manager by default, so that Lucene JUL logs are included with Solr logs.
+LOG4J_CONFIG=("-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager")
Review Comment:
No, `solr.in.sh` cannot override this. But good call on windows, will add that in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1629765785
Looks fine on main branch!
I just noticed: we still have "-XX:CompileCommand=exclude,com.github.benmanes.caffeine.cache.BoundedLocalCache::put" in the command line options. Wasn't this bug fixed? Before opening another issue I ask here....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on a diff in pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #1765:
URL: https://github.com/apache/solr/pull/1765#discussion_r1257489646
##########
solr/bin/solr:
##########
@@ -1581,7 +1581,8 @@ if [ "${SOLR_HOME:0:${#EXAMPLE_DIR}}" = "$EXAMPLE_DIR" ]; then
SOLR_LOGS_DIR="$SOLR_HOME/../logs"
fi
-LOG4J_CONFIG=()
+# Set the logging manager by default, so that Lucene JUL logs are included with Solr logs.
+LOG4J_CONFIG=("-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager")
Review Comment:
It looks like this is misisng for Windows `solr.cmd`.
Small question: I hope this is an internal variable that is not overwritten by user code in `solr.in.sh`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1625992983
Unfortunately `MethodHandles.lookup().ensureInitialited(VectorUtil.class)` is only allowed in Java 15+, and Solr is on Java 11.
We could always just do a quick:
```java
VectorUtil.dotProduct(new byte[0], new byte[0]);
```
This works, so will probably commit separately. Do we need to be worried about the security manager stuff, or will the lucene hack work around it for us?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1765:
URL: https://github.com/apache/solr/pull/1765#discussion_r1258324388
##########
solr/bin/solr:
##########
@@ -1581,7 +1581,8 @@ if [ "${SOLR_HOME:0:${#EXAMPLE_DIR}}" = "$EXAMPLE_DIR" ]; then
SOLR_LOGS_DIR="$SOLR_HOME/../logs"
fi
-LOG4J_CONFIG=()
+# Set the logging manager by default, so that Lucene JUL logs are included with Solr logs.
+LOG4J_CONFIG=("-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager")
Review Comment:
Ok it should work for windows too, but it does it in a slightly different way. Shouldn't be a big deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1629497333
It also shows the log levels here:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1629769482
Ignore this, I asked question on #1090.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1629045722
On startup I now see:
```
2023-07-10 14:06:29.033 INFO (main) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 20.0.1+9
2023-07-10 14:06:29.159 WARN (main) [] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2023-07-10 14:06:29.169 INFO (main) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2023-07-10 14:06:29.171 INFO (main) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 10.0.0-SNAPSHOT 387817d1c2f8f38e03dc58456cc0dd5f4b9c56b5 [snapshot build, details omitted]
2023-07-10 14:06:29.171 INFO (main) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in standalone mode on port 8983
2023-07-10 14:06:29.171 INFO (main) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /Users/houstonputman/dev/oss/solr/solr/main/solr/packaging/build/dev
2023-07-10 14:06:29.171 INFO (main) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2023-07-10T14:06:29.171952Z
2023-07-10 14:06:29.173 INFO (main) [] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /Users/houstonputman/dev/oss/solr/solr/main/solr/packaging/build/dev/example/films/solr/../logs/jvm_crash_27222.log
2023-07-10 14:06:29.177 INFO (main) [] o.a.l.s.MemorySegmentIndexInputProvider Using MemorySegmentIndexInput with Java 20; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2023-07-10 14:06:29.194 INFO (main) [] o.a.l.u.VectorUtilPanamaProvider Java vector incubator API enabled; uses preferredBitSize=128
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1627728458
To make sure that all logs are written on early startup, we should add this code to the early startup phase after logging is fully configured:
```java
Stream.of(MMapDirectory.class, VectorUtil.class).forEach(cls -> {
try {
Class.forName(cls.getName());
} catch (ReflectiveOperationException re) {
throw new SolrException(...)
}
});
```
Maybe add a TODO to use `MethodHandles.lookup().ensureClassInitialized` in Solr 10 w/ Java 17:
```java
final var lookup = MethodHandles.lookup();
Stream.of(MMapDirectory.class, VectorUtil.class).forEach(cls -> {
try {
lookup.ensureInitialized(cls);
} catch (ReflectiveOperationException re) {
throw new SolrException(...)
}
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1765:
URL: https://github.com/apache/solr/pull/1765#discussion_r1258715694
##########
solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java:
##########
@@ -226,6 +229,21 @@ public void init(ServletContext servletContext) {
StartupLoggingUtils.changeLogLevel(logLevel);
}
+ // Do initial logs for experimental Lucene classes.
+ // TODO: Use "MethodHandles.lookup().ensureClassInitialized()" instead of "Class.forName()"
+ // once JDK 15+ is mandatory
+ Stream.of(MMapDirectory.class, VectorUtil.class)
+ .forEach(
+ cls -> {
+ try {
+ Class.forName(cls.getName());
+ } catch (ReflectiveOperationException re) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "Could not load Lucene class: " + cls.getName());
+ }
+ });
+ VectorUtil.dotProduct(new byte[0], new byte[0]);
Review Comment:
My bad, I thought I had deleted that. Will remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman merged pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman merged PR #1765:
URL: https://github.com/apache/solr/pull/1765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] uschindler commented on pull request #1765: SOLR-16878: Add JUL Log4J manager for lucene logs
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #1765:
URL: https://github.com/apache/solr/pull/1765#issuecomment-1626256875
The alternative is: `Class.forName(VectorUtil.class.getName())`
Security Manager is fine for us as Lucene wraps it. Solr has no components that run with zero privileges like Elasticsearch script engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org