You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@solr.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2026/06/30 12:50:03 UTC
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 249 - Still Failing!
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/249/
No tests ran.
Build Log:
[...truncated 360 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-30 12:25:19.784; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:19.789; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:20.278; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:20.278; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1715 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38671 is already being used by another process (pid: 3840819)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-30 12:25:21.482; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:21.485; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 18410 ms
ok 11 package with basic auth # in 12774 ms
ok 12 admin UI static content is excepted from basic auth # in 11157 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 167 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 81 ms
ok 16 compressed response can be decompressed and parsed # in 82 ms
ok 17 setting property # in 11443 ms
ok 18 short form of setting property # in 6588 ms
ok 19 connecting to solr via various connection types # in 11003 ms
ok 20 create for non cloud mode # in 8074 ms
ok 21 create for cloud mode # in 10348 ms
ok 22 multiple connection options are prevented # in 909 ms
ok 23 create collection # in 7952 ms
ok 24 create collection using solrUrl # in 5369 ms
ok 25 create collection using legacy solrUrl # in 5479 ms
ok 26 create collection using Zookeeper # in 5559 ms
ok 27 reject d option with invalid config dir # in 2810 ms
ok 28 accept d option with builtin config # in 5443 ms
ok 29 accept d option with explicit path to config # in 5394 ms
ok 30 accept n option as config name # in 5374 ms
ok 31 allow config reuse when n option specifies same config # in 9414 ms
ok 32 create multisharded collections when s provided # in 5368 ms
ok 33 create replicated collections when rf provided # in 6246 ms
ok 34 can delete collections # in 11434 ms
ok 35 can delete collections with solr-url # in 8493 ms
ok 36 can delete collections with solr-connection # in 8364 ms
ok 37 collection delete also deletes zk config # in 7642 ms
ok 38 deletes accompanying zk config with nondefault name # in 7526 ms
ok 39 delete-config option can opt to leave config in zk # in 7359 ms
ok 40 start -e cloud works with --jvm-opts # in 21994 ms
ok 41 start -e cloud works with --no-prompt # in 43838 ms
ok 42 Check export command # in 19564 ms
ok 43 export fails on non cloud mode # in 9740 ms
Waiting for Tika Server to be ready on port 38676
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16639 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16593 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16614 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 10744 ms
ok 48 healthcheck on remote cloud solr # in 10616 ms
ok 49 healthcheck errors on standalone solr # in 11622 ms
ok 50 solr help flag prints help # in 755 ms
ok 51 solr with no flags prints help # in 776 ms
ok 52 start help flag prints help # in 160 ms
ok 53 start h flag prints help # in 170 ms
ok 54 stop help flag prints help # in 183 ms
ok 55 restart help flag prints help # in 149 ms
ok 56 status help flag prints help # in 902 ms
ok 57 healthcheck help flag prints help # in 779 ms
ok 58 create help flag prints help # in 789 ms
ok 59 delete help flag prints help # in 846 ms
ok 60 zk help flag prints help # in 832 ms
ok 61 auth help flag prints help # in 807 ms
ok 62 assert help flag prints help # in 810 ms
ok 63 post help flag prints help # in 798 ms
ok 64 SQL Module # in 16163 ms
ok 65 icu collation in analysis-extras module # in 13091 ms
ok 66 Check lifecycle of sentiment classification # in 23696 ms
ok 67 lifecycle of package # in 8572 ms
ok 68 deploying and undeploying a collection level package # in 13671 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 16327 ms
ok 71 Random placement plugin using ENV # in 16180 ms
ok 72 Check help command # in 2440 ms
ok 73 basic post with a type specified # in 5521 ms
ok 74 basic post with collection instead of url parameter # in 4076 ms
ok 75 basic post with solr-url and collection # in 4013 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 4052 ms
ok 77 basic post WITHOUT a type specified # in 11873 ms
ok 78 crawling a directory as a dry-run # in 790 ms
ok 79 crawling a directory # in 4499 ms
ok 80 skipcommit and optimize and delete # in 4070 ms
ok 81 args mode # in 7133 ms
ok 82 stdin mode # in 3525 ms
ok 83 verbose echo the Solr response # in 3909 ms
ok 84 post solr log into solr via cli # in 9887 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 27 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-30T12:35:54.550909185Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-30T12:35:54.550951947Z It should be set to 65000 to avoid operational disruption.
2026-06-30T12:35:54.550955008Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-30T12:35:54.551258270Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-30T12:35:54.551288184Z
2026-06-30T12:35:54.564941461Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-30T12:35:54.575824741Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-30T12:35:54.640864625Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-30T12:35:54.690288316Z WARNING: A command line option has enabled the Security Manager
2026-06-30T12:35:54.690316937Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-30T12:35:57.256469140Z 2026-06-30 12:35:57.250 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-30T12:35:57.804325541Z 2026-06-30 12:35:57.803 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-30T12:35:57.855314415Z 2026-06-30 12:35:57.854 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-30T12:35:57.917530071Z 2026-06-30 12:35:57.916 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-30T12:35:57.917897726Z 2026-06-30 12:35:57.917 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-30T12:35:57.917947402Z 2026-06-30 12:35:57.917 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-30T12:35:57.917955776Z 2026-06-30 12:35:57.917 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-30T12:35:57.917291391Z
2026-06-30T12:35:57.921813618Z 2026-06-30 12:35:57.921 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-30T12:35:58.017549573Z 2026-06-30 12:35:58.016 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-30T12:35:58.117251670Z 2026-06-30 12:35:58.116 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-30T12:35:58.117273064Z 2026-06-30 12:35:58.116 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-30T12:35:58.225009411Z 2026-06-30 12:35:58.224 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-30T12:35:59.836348789Z 2026-06-30 12:35:59.835 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-30T12:36:00.343050701Z 2026-06-30 12:36:00.342 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-30T12:36:00.356538268Z 2026-06-30 12:36:00.356 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-30T12:36:00.356586392Z 2026-06-30 12:36:00.356 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-30T12:36:00.362842261Z 2026-06-30 12:36:00.362 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-30T12:36:00.362860204Z 2026-06-30 12:36:00.362 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-30T12:36:00.451185734Z 2026-06-30 12:36:00.450 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-30T12:36:00.865523017Z 2026-06-30 12:36:00.864 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-30T12:36:00.895291722Z 2026-06-30 12:36:00.894 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:00.952192320Z 2026-06-30 12:36:00.951 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-30T12:36:00.981591030Z 2026-06-30 12:36:00.980 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:00.981614125Z 2026-06-30 12:36:00.981 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:01.116294798Z 2026-06-30 12:36:01.115 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-30T12:36:01.144317758Z 2026-06-30 12:36:01.143 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:01.156305071Z 2026-06-30 12:36:01.155 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-30T12:36:01.163157173Z 2026-06-30 12:36:01.162 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72211905013022720-solr-node1:8983_solr-n_0000000000) starting
2026-06-30T12:36:01.256719268Z 2026-06-30 12:36:01.256 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-30T12:36:01.258241836Z 2026-06-30 12:36:01.257 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-30T12:36:01.269882857Z 2026-06-30 12:36:01.269 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-30T12:36:01.339563200Z 2026-06-30 12:36:01.338 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-30T12:36:02.098931877Z 2026-06-30 12:36:02.098 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-30T12:36:02.448914594Z 2026-06-30 12:36:02.448 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-30T12:36:02.987660217Z 2026-06-30 12:36:02.987 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:03.474744324Z 2026-06-30 12:36:03.473 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-30T12:36:03.537341806Z 2026-06-30 12:36:03.536 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_30.request.log
2026-06-30T12:36:03.544208562Z 2026-06-30 12:36:03.543 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-30T12:36:03.546358308Z 2026-06-30 12:36:03.545 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @8971ms
2026-06-30T12:36:04.059633144Z 2026-06-30 12:36:04.059 INFO (qtp985324122-28-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=185
2026-06-30T12:36:04.391558628Z 2026-06-30 12:36:04.391 INFO (qtp985324122-97-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19
2026-06-30T12:36:09.266627422Z 2026-06-30 12:36:09.266 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-30T12:36:09.873312912Z 2026-06-30 12:36:09.872 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-30T12:36:16.156824845Z 2026-06-30 12:36:16.155 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-06-30T12:36:16.250284588Z 2026-06-30 12:36:16.249 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=33
2026-06-30T12:36:16.287984776Z 2026-06-30 12:36:16.287 INFO (qtp985324122-28-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=3
2026-06-30T12:36:16.769002614Z 2026-06-30 12:36:16.768 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-30T12:36:16.808289161Z 2026-06-30 12:36:16.807 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-30T12:36:16.826741457Z 2026-06-30 12:36:16.826 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:16.941832654Z 2026-06-30 12:36:16.941 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-30T12:36:16.941859648Z "core":"test-collection_shard3_replica_n1",
2026-06-30T12:36:16.941863040Z "node_name":"solr-node1:8983_solr",
2026-06-30T12:36:16.941865492Z "base_url":"http://solr-node1:8983/solr",
2026-06-30T12:36:16.941867792Z "collection":"test-collection",
2026-06-30T12:36:16.941870315Z "shard":"shard3",
2026-06-30T12:36:16.941872391Z "state":"down",
2026-06-30T12:36:16.941874441Z "type":"NRT",
2026-06-30T12:36:16.941876437Z "operation":"ADDREPLICA",
2026-06-30T12:36:16.941879022Z "waitForFinalState":"false"}
2026-06-30T12:36:16.955543252Z 2026-06-30 12:36:16.954 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-30T12:36:16.955571028Z "core":"test-collection_shard2_replica_n2",
2026-06-30T12:36:16.955574050Z "node_name":"solr-node3:8985_solr",
2026-06-30T12:36:16.955577393Z "base_url":"http://solr-node3:8985/solr",
2026-06-30T12:36:16.955579678Z "collection":"test-collection",
2026-06-30T12:36:16.955581761Z "shard":"shard2",
2026-06-30T12:36:16.955583755Z "state":"down",
2026-06-30T12:36:16.955585944Z "type":"NRT",
2026-06-30T12:36:16.955587823Z "operation":"ADDREPLICA",
2026-06-30T12:36:16.955589757Z "waitForFinalState":"false"}
2026-06-30T12:36:16.964320064Z 2026-06-30 12:36:16.963 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-30T12:36:16.964344747Z "core":"test-collection_shard1_replica_n4",
2026-06-30T12:36:16.964347602Z "node_name":"solr-node2:8984_solr",
2026-06-30T12:36:16.964364080Z "base_url":"http://solr-node2:8984/solr",
2026-06-30T12:36:16.964366345Z "collection":"test-collection",
2026-06-30T12:36:16.964368350Z "shard":"shard1",
2026-06-30T12:36:16.964370283Z "state":"down",
2026-06-30T12:36:16.964372276Z "type":"NRT",
2026-06-30T12:36:16.964374313Z "operation":"ADDREPLICA",
2026-06-30T12:36:16.964376251Z "waitForFinalState":"false"}
2026-06-30T12:36:17.078848605Z 2026-06-30 12:36:17.078 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:17.224735665Z 2026-06-30 12:36:17.224 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-30T12:36:17.385039436Z 2026-06-30 12:36:17.384 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-30T12:36:17.524382511Z 2026-06-30 12:36:17.522 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-30T12:36:18.239618842Z 2026-06-30 12:36:18.238 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-30T12:36:18.239648283Z 2026-06-30 12:36:18.239 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-30T12:36:18.301296188Z 2026-06-30 12:36:18.300 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-06-30T12:36:19.768834907Z 2026-06-30 12:36:19.767 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:19.836794016Z 2026-06-30 12:36:19.836 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-30T12:36:19.841868799Z 2026-06-30 12:36:19.840 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-30T12:36:19.842778426Z 2026-06-30 12:36:19.841 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-30T12:36:19.842946327Z 2026-06-30 12:36:19.842 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-30T12:36:19.915598214Z 2026-06-30 12:36:19.915 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-30T12:36:19.922732992Z 2026-06-30 12:36:19.922 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-30T12:36:19.924832029Z 2026-06-30 12:36:19.924 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-30T12:36:19.926637572Z 2026-06-30 12:36:19.926 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-30T12:36:19.950377724Z 2026-06-30 12:36:19.949 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-30T12:36:19.975689865Z 2026-06-30 12:36:19.975 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-30T12:36:19.993646315Z 2026-06-30 12:36:19.993 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-30T12:36:19.993673597Z 2026-06-30 12:36:19.993 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-30T12:36:20.002596419Z 2026-06-30 12:36:20.002 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-30T12:36:20.012026938Z 2026-06-30 12:36:20.011 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-30T12:36:20.016318963Z 2026-06-30 12:36:20.015 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-30T12:36:20.033276793Z 2026-06-30 12:36:20.032 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-30T12:36:20.033299248Z 2026-06-30 12:36:20.033 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-30T12:36:20.033355758Z 2026-06-30 12:36:20.033 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-06-30T12:36:20.034866722Z 2026-06-30 12:36:20.034 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-30T12:36:20.035350675Z 2026-06-30 12:36:20.035 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-06-30T12:36:20.036542316Z 2026-06-30 12:36:20.036 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72211905013022720-core_node3-n_0000000000
2026-06-30T12:36:20.041638624Z 2026-06-30 12:36:20.041 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-06-30T12:36:20.148181385Z 2026-06-30 12:36:20.147 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.152110319Z 2026-06-30 12:36:20.151 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-30T12:36:20.159275633Z 2026-06-30 12:36:20.158 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2936
2026-06-30T12:36:20.260022506Z 2026-06-30 12:36:20.259 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3874439,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-30T12:35:52.85964948Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-30T12:36:05.352428393Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-30T12:36:05.352474801Z It should be set to 65000 to avoid operational disruption.
2026-06-30T12:36:05.352480161Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-30T12:36:05.352731650Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-30T12:36:05.352739751Z
2026-06-30T12:36:05.362455677Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-30T12:36:05.372009841Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-30T12:36:05.424723388Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-30T12:36:05.462127653Z WARNING: A command line option has enabled the Security Manager
2026-06-30T12:36:05.462173345Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-30T12:36:07.663109673Z 2026-06-30 12:36:07.659 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-30T12:36:07.965196332Z 2026-06-30 12:36:07.964 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-30T12:36:07.995956641Z 2026-06-30 12:36:07.995 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-30T12:36:08.005144306Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-30T12:36:08.005518241Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-30T12:36:08.005526899Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-30T12:36:08.005530641Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-30T12:36:08.004959307Z
2026-06-30T12:36:08.009050609Z 2026-06-30 12:36:08.008 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-30T12:36:08.034199901Z 2026-06-30 12:36:08.033 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-30T12:36:08.071472978Z 2026-06-30 12:36:08.070 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:08.357547855Z 2026-06-30 12:36:08.357 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-30T12:36:08.382666565Z 2026-06-30 12:36:08.382 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:08.382700753Z 2026-06-30 12:36:08.382 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:08.382822804Z 2026-06-30 12:36:08.382 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:08.493201133Z 2026-06-30 12:36:08.492 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-30T12:36:08.493350081Z 2026-06-30 12:36:08.493 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-30T12:36:08.493501086Z 2026-06-30 12:36:08.493 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-30T12:36:08.530009957Z 2026-06-30 12:36:08.529 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-30T12:36:09.586780668Z 2026-06-30 12:36:09.586 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-30T12:36:09.743722871Z 2026-06-30 12:36:09.743 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-30T12:36:09.745825393Z 2026-06-30 12:36:09.743 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-30T12:36:09.754710710Z 2026-06-30 12:36:09.754 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.760131196Z 2026-06-30 12:36:09.759 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-30T12:36:09.764446794Z 2026-06-30 12:36:09.764 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:09.764641550Z 2026-06-30 12:36:09.764 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:09.800962873Z 2026-06-30 12:36:09.800 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-30T12:36:09.809582523Z 2026-06-30 12:36:09.809 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-30T12:36:09.836268950Z 2026-06-30 12:36:09.835 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.848651945Z 2026-06-30 12:36:09.848 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-30T12:36:09.864199171Z 2026-06-30 12:36:09.863 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-30T12:36:09.874498816Z 2026-06-30 12:36:09.874 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-30T12:36:09.928701897Z 2026-06-30 12:36:09.928 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-30T12:36:10.210828671Z 2026-06-30 12:36:10.210 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-30T12:36:10.399466477Z 2026-06-30 12:36:10.398 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-30T12:36:10.667609046Z 2026-06-30 12:36:10.667 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:10.794966884Z 2026-06-30 12:36:10.794 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-30T12:36:10.800969044Z 2026-06-30 12:36:10.800 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_30.request.log
2026-06-30T12:36:10.804461011Z 2026-06-30 12:36:10.803 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-30T12:36:10.805258608Z 2026-06-30 12:36:10.805 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5434ms
2026-06-30T12:36:11.030462368Z 2026-06-30 12:36:11.029 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=58
2026-06-30T12:36:11.160001881Z 2026-06-30 12:36:11.159 INFO (qtp985324122-56-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-06-30T12:36:17.224299106Z 2026-06-30 12:36:17.223 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-30T12:36:17.291832663Z 2026-06-30 12:36:17.289 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-30T12:36:17.401378993Z 2026-06-30 12:36:17.400 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-30T12:36:17.917359349Z 2026-06-30 12:36:17.916 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-30T12:36:17.917394256Z 2026-06-30 12:36:17.916 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-30T12:36:17.949569753Z 2026-06-30 12:36:17.949 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-30T12:36:18.729512404Z 2026-06-30 12:36:18.728 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:18.755532434Z 2026-06-30 12:36:18.755 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-30T12:36:18.770900992Z 2026-06-30 12:36:18.770 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-30T12:36:18.772301223Z 2026-06-30 12:36:18.771 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-30T12:36:18.772375185Z 2026-06-30 12:36:18.772 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-30T12:36:18.836414113Z 2026-06-30 12:36:18.835 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-30T12:36:18.840540057Z 2026-06-30 12:36:18.840 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-30T12:36:18.843240280Z 2026-06-30 12:36:18.842 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-30T12:36:18.845993119Z 2026-06-30 12:36:18.845 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-30T12:36:18.867526846Z 2026-06-30 12:36:18.866 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-30T12:36:18.877353482Z 2026-06-30 12:36:18.876 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-30T12:36:18.885125923Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-30T12:36:18.885148125Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-30T12:36:18.906970171Z 2026-06-30 12:36:18.906 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-30T12:36:18.930691200Z 2026-06-30 12:36:18.929 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-30T12:36:18.932400239Z 2026-06-30 12:36:18.932 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-30T12:36:18.950225987Z 2026-06-30 12:36:18.949 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-30T12:36:18.950286988Z 2026-06-30 12:36:18.949 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-30T12:36:18.950351077Z 2026-06-30 12:36:18.950 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-30T12:36:18.952272822Z 2026-06-30 12:36:18.951 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-30T12:36:18.953258334Z 2026-06-30 12:36:18.953 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-30T12:36:18.954926918Z 2026-06-30 12:36:18.954 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72211905013022724-core_node6-n_0000000000
2026-06-30T12:36:18.973141714Z 2026-06-30 12:36:18.972 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-30T12:36:19.083681932Z 2026-06-30 12:36:19.083 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:19.089548646Z 2026-06-30 12:36:19.089 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-30T12:36:19.103633828Z 2026-06-30 12:36:19.103 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1881
2026-06-30T12:36:19.206266158Z 2026-06-30 12:36:19.205 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.148495885Z 2026-06-30 12:36:20.147 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.259948258Z 2026-06-30 12:36:20.259 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3874843,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-30T12:36:04.796102086Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-30T12:36:05.352697068Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-30T12:36:05.352722043Z It should be set to 65000 to avoid operational disruption.
2026-06-30T12:36:05.352725204Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-30T12:36:05.353083003Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-30T12:36:05.353095323Z
2026-06-30T12:36:05.362672573Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-30T12:36:05.371923394Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-30T12:36:05.424665619Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-30T12:36:05.462181551Z WARNING: A command line option has enabled the Security Manager
2026-06-30T12:36:05.462202669Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-30T12:36:07.018495886Z 2026-06-30 12:36:07.014 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-30T12:36:07.338992303Z 2026-06-30 12:36:07.338 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-30T12:36:07.371255765Z 2026-06-30 12:36:07.370 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-30T12:36:07.381430818Z 2026-06-30 12:36:07.380 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-30T12:36:07.381501108Z 2026-06-30 12:36:07.381 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-30T12:36:07.381621174Z 2026-06-30 12:36:07.381 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-30T12:36:07.381726936Z 2026-06-30 12:36:07.381 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-30T12:36:07.381199368Z
2026-06-30T12:36:07.384783909Z 2026-06-30 12:36:07.384 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-30T12:36:07.421295016Z 2026-06-30 12:36:07.420 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-30T12:36:07.489356333Z 2026-06-30 12:36:07.488 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:07.663368557Z 2026-06-30 12:36:07.662 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-30T12:36:07.684918498Z 2026-06-30 12:36:07.684 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:07.685024769Z 2026-06-30 12:36:07.684 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:07.685162403Z 2026-06-30 12:36:07.684 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:07.800958684Z 2026-06-30 12:36:07.800 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-30T12:36:07.802034867Z 2026-06-30 12:36:07.800 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-30T12:36:07.802235307Z 2026-06-30 12:36:07.801 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-30T12:36:07.856849195Z 2026-06-30 12:36:07.856 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-30T12:36:08.899447678Z 2026-06-30 12:36:08.898 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-30T12:36:09.123366398Z 2026-06-30 12:36:09.122 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-30T12:36:09.125436362Z 2026-06-30 12:36:09.122 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-30T12:36:09.138662035Z 2026-06-30 12:36:09.138 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.146377911Z 2026-06-30 12:36:09.145 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-30T12:36:09.149916037Z 2026-06-30 12:36:09.149 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:09.150133790Z 2026-06-30 12:36:09.149 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:09.189254720Z 2026-06-30 12:36:09.188 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-30T12:36:09.197620347Z 2026-06-30 12:36:09.197 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-30T12:36:09.231503799Z 2026-06-30 12:36:09.230 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.243707563Z 2026-06-30 12:36:09.243 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-30T12:36:09.255457775Z 2026-06-30 12:36:09.254 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-30T12:36:09.267734741Z 2026-06-30 12:36:09.267 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-30T12:36:09.335603339Z 2026-06-30 12:36:09.334 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-30T12:36:09.655132527Z 2026-06-30 12:36:09.654 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-30T12:36:09.873642195Z 2026-06-30 12:36:09.873 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-30T12:36:09.891950947Z 2026-06-30 12:36:09.891 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-30T12:36:10.217638422Z 2026-06-30 12:36:10.217 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:10.368035789Z 2026-06-30 12:36:10.367 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-30T12:36:10.375329305Z 2026-06-30 12:36:10.374 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_30.request.log
2026-06-30T12:36:10.381266526Z 2026-06-30 12:36:10.380 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-30T12:36:10.382174004Z 2026-06-30 12:36:10.381 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5011ms
2026-06-30T12:36:13.860142370Z 2026-06-30 12:36:13.855 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=116
2026-06-30T12:36:13.962891801Z 2026-06-30 12:36:13.962 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=22
2026-06-30T12:36:16.718824764Z 2026-06-30 12:36:16.718 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=10
2026-06-30T12:36:17.224004464Z 2026-06-30 12:36:17.222 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-30T12:36:17.301565699Z 2026-06-30 12:36:17.300 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-30T12:36:17.408899788Z 2026-06-30 12:36:17.408 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-30T12:36:17.919108397Z 2026-06-30 12:36:17.918 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-30T12:36:17.919157776Z 2026-06-30 12:36:17.918 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-30T12:36:17.949576739Z 2026-06-30 12:36:17.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-06-30T12:36:18.729459412Z 2026-06-30 12:36:18.728 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:18.755590987Z 2026-06-30 12:36:18.755 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-30T12:36:18.770904133Z 2026-06-30 12:36:18.770 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-30T12:36:18.772104995Z 2026-06-30 12:36:18.771 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-30T12:36:18.772172428Z 2026-06-30 12:36:18.771 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-30T12:36:18.836413094Z 2026-06-30 12:36:18.835 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-30T12:36:18.840415249Z 2026-06-30 12:36:18.839 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-30T12:36:18.844296865Z 2026-06-30 12:36:18.843 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-30T12:36:18.855184622Z 2026-06-30 12:36:18.854 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-30T12:36:18.870195349Z 2026-06-30 12:36:18.869 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-30T12:36:18.879038416Z 2026-06-30 12:36:18.878 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-30T12:36:18.885164852Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-30T12:36:18.885244874Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-30T12:36:18.907288751Z 2026-06-30 12:36:18.906 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-30T12:36:18.930588718Z 2026-06-30 12:36:18.929 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-30T12:36:18.932238116Z 2026-06-30 12:36:18.931 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-30T12:36:18.949747160Z 2026-06-30 12:36:18.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-30T12:36:18.949776009Z 2026-06-30 12:36:18.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-30T12:36:18.949793068Z 2026-06-30 12:36:18.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-06-30T12:36:18.951812284Z 2026-06-30 12:36:18.951 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-30T12:36:18.952369594Z 2026-06-30 12:36:18.952 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-06-30T12:36:18.954284507Z 2026-06-30 12:36:18.954 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72211905013022723-core_node5-n_0000000000
2026-06-30T12:36:18.975646137Z 2026-06-30 12:36:18.975 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-06-30T12:36:19.083723174Z 2026-06-30 12:36:19.083 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:19.089161179Z 2026-06-30 12:36:19.088 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-30T12:36:19.102369616Z 2026-06-30 12:36:19.101 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1880
2026-06-30T12:36:19.206448907Z 2026-06-30 12:36:19.205 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.148296989Z 2026-06-30 12:36:20.147 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.173438126Z 2026-06-30 12:36:20.172 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-30T12:36:20.260054617Z 2026-06-30 12:36:20.259 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.262618990Z 2026-06-30 12:36:20.262 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3522
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3874970,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-30T12:36:05.001479307Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 31169 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Image is up to date for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 649b7c42f74ea036f61c7cf952c066c83501dd51fb60557609b30eea9737c0a3
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# a0f85b1c1772d934d438563f26b5e9faa9baa2ae57025c13b15db1a97fd1539f
# Starting solr-node2 connected to first node's ZooKeeper
# 069309e2a2e2dab82a50fc0d975525773d88c7995f809dd0a189ce0c46de2e09
# Started solr-node3 connected to first node's ZooKeeper
# bdaa1dfa287b895579204a77d08082948b265a849c3b5f2cc4d6ad98b5524a64
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-30 12:36:16.355; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:36:16.438; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 16371 ms
ok 87 snapshot lifecycle # in 5043 ms
ok 88 snapshot list # in 3434 ms
ok 89 snapshot list via --solr-connection # in 3467 ms
ok 90 snapshot describe # in 3506 ms
ok 91 snapshot describe via --solr-connection # in 3619 ms
ok 92 start solr with ssl # in 17316 ms
ok 93 use different hostname when not checking peer-name # in 25267 ms
ok 94 start solr with ssl and auth # in 15169 ms
ok 95 start solr with client truststore and security manager # in 17086 ms
ok 96 start solr with mTLS needed # in 47267 ms
ok 97 start solr with mTLS wanted # in 35975 ms
ok 98 test keystore reload # in 60702 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 16316 ms
ok 100 stop command for single port # in 18749 ms
ok 101 check stop command doesn't hang # in 225 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8310 ms
ok 103 deprecated system properties converted to modern properties # in 195424 ms
ok 104 start with custom jetty options # in 8138 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 9133 ms
ok 106 bootstrapping a configset # in 9544 ms
ok 107 status detects locally running solr # in 9361 ms
ok 108 status with --solr-url from user # in 8075 ms
ok 109 status with --port from user # in 8034 ms
ok 110 multiple connection options are prevented # in 850 ms
ok 111 status with invalid --solr-url from user # in 1237 ms
ok 112 status with --short format # in 195837 ms
ok 113 searching solr via locally executed streaming expression # in 1741 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1507 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1197 ms
ok 116 variable interpolation # in 1751 ms
ok 117 searching solr without credentials fails with error # in 1164 ms
ok 118 --version returns Solr version # in 731 ms
ok 119 version as direct tool call still runs # in 729 ms
ok 120 short help # in 752 ms
ok 121 short help is inferred # in 763 ms
ok 122 long help # in 768 ms
ok 123 running subcommands with zk is prevented # in 740 ms
ok 124 listing out files # in 2129 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 9191 ms
ok 126 copying files around # in 10834 ms
ok 127 upconfig # in 2566 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1154 ms
ok 129 downconfig # in 1191 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1451 ms
ok 131 env var ZK_HOST is honored # in 3173 ms
ok 132 Cleanup # in 1844 ms
> Task :solr:packaging:integrationTests FAILED
[Incubating] Problems report is available at: file:///home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests' (registered in build file 'solr/packaging/build.gradle').
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/9.6.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 35s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/wtd3utwi3v6hk
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 251 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/251/
No tests ran.
Build Log:
[...truncated 350 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-07-02 14:31:15.001; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:15.004; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:15.472; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:15.472; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1314 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 44869 is already being used by another process (pid: 1668090)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-07-02 14:31:16.641; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:16.644; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 17570 ms
ok 11 package with basic auth # in 12254 ms
ok 12 admin UI static content is excepted from basic auth # in 10586 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 144 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 81 ms
ok 16 compressed response can be decompressed and parsed # in 78 ms
ok 17 setting property # in 10995 ms
ok 18 short form of setting property # in 6461 ms
ok 19 connecting to solr via various connection types # in 10503 ms
ok 20 create for non cloud mode # in 8075 ms
ok 21 create for cloud mode # in 10187 ms
ok 22 multiple connection options are prevented # in 869 ms
ok 23 create collection # in 7815 ms
ok 24 create collection using solrUrl # in 5242 ms
ok 25 create collection using legacy solrUrl # in 5361 ms
ok 26 create collection using Zookeeper # in 5508 ms
ok 27 reject d option with invalid config dir # in 2714 ms
ok 28 accept d option with builtin config # in 5322 ms
ok 29 accept d option with explicit path to config # in 5285 ms
ok 30 accept n option as config name # in 5286 ms
ok 31 allow config reuse when n option specifies same config # in 9163 ms
ok 32 create multisharded collections when s provided # in 5321 ms
ok 33 create replicated collections when rf provided # in 6149 ms
ok 34 can delete collections # in 10977 ms
ok 35 can delete collections with solr-url # in 8562 ms
ok 36 can delete collections with solr-connection # in 8062 ms
ok 37 collection delete also deletes zk config # in 7409 ms
ok 38 deletes accompanying zk config with nondefault name # in 7317 ms
ok 39 delete-config option can opt to leave config in zk # in 7150 ms
ok 40 start -e cloud works with --no-prompt # in 2705 ms
ok 41 start -e cloud works with --jvm-opts # in 106 ms
ok 42 Check export command # in 18993 ms
ok 43 export fails on non cloud mode # in 9613 ms
Waiting for Tika Server to be ready on port 44874
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16574 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16538 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16543 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 13349 ms
ok 48 healthcheck errors on standalone solr # in 6811 ms
ok 49 solr help flag prints help # in 752 ms
ok 50 solr with no flags prints help # in 740 ms
ok 51 start help flag prints help # in 147 ms
ok 52 start h flag prints help # in 143 ms
ok 53 stop help flag prints help # in 148 ms
ok 54 restart help flag prints help # in 144 ms
ok 55 status help flag prints help # in 788 ms
ok 56 healthcheck help flag prints help # in 753 ms
ok 57 create help flag prints help # in 751 ms
ok 58 delete help flag prints help # in 812 ms
ok 59 zk help flag prints help # in 766 ms
ok 60 auth help flag prints help # in 761 ms
ok 61 assert help flag prints help # in 754 ms
ok 62 post help flag prints help # in 779 ms
ok 63 SQL Module # in 16272 ms
ok 64 icu collation in analysis-extras module # in 12534 ms
ok 65 Check lifecycle of sentiment classification # in 23561 ms
ok 66 Affinity placement plugin using sysprop # in 15839 ms
ok 67 Random placement plugin using ENV # in 15305 ms
ok 68 Check help command # in 2316 ms
ok 69 basic post with a type specified # in 5487 ms
ok 70 basic post with collection instead of url parameter # in 3964 ms
ok 71 basic post with solr-url and collection # in 3962 ms
ok 72 basic post with -s (solr-connection short form) and collection # in 3932 ms
ok 73 basic post WITHOUT a type specified # in 11294 ms
ok 74 crawling a directory as a dry-run # in 763 ms
ok 75 crawling a directory # in 4294 ms
ok 76 skipcommit and optimize and delete # in 3784 ms
ok 77 args mode # in 6848 ms
ok 78 stdin mode # in 3407 ms
ok 79 verbose echo the Solr response # in 3694 ms
ok 80 post solr log into solr via cli # in 9744 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 25 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-07-02T14:40:19.606527161Z *** [WARN] *** Your open file limit is currently 1024.
2026-07-02T14:40:19.606572796Z It should be set to 65000 to avoid operational disruption.
2026-07-02T14:40:19.606577499Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-07-02T14:40:19.606830320Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-07-02T14:40:19.606833549Z
2026-07-02T14:40:19.623828557Z Java 17 detected. Enabled workaround for SOLR-16463
2026-07-02T14:40:19.636344550Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-07-02T14:40:19.692223525Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-07-02T14:40:19.739373337Z WARNING: A command line option has enabled the Security Manager
2026-07-02T14:40:19.739403993Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-07-02T14:40:21.923545647Z 2026-07-02 14:40:21.918 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-07-02T14:40:22.459142044Z 2026-07-02 14:40:22.458 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-07-02T14:40:22.511966025Z 2026-07-02 14:40:22.511 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-07-02T14:40:22.525781070Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-07-02T14:40:22.525850140Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-07-02T14:40:22.526079965Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-07-02T14:40:22.526086653Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-07-02T14:40:22.525548069Z
2026-07-02T14:40:22.529902728Z 2026-07-02 14:40:22.529 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-07-02T14:40:22.575286985Z 2026-07-02 14:40:22.574 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-07-02T14:40:22.624536554Z 2026-07-02 14:40:22.624 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-07-02T14:40:22.624652769Z 2026-07-02 14:40:22.624 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-07-02T14:40:22.714572874Z 2026-07-02 14:40:22.713 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-07-02T14:40:24.178740445Z 2026-07-02 14:40:24.178 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-07-02T14:40:24.645719265Z 2026-07-02 14:40:24.645 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-07-02T14:40:24.658455148Z 2026-07-02 14:40:24.657 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-07-02T14:40:24.658478216Z 2026-07-02 14:40:24.657 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-07-02T14:40:24.663425254Z 2026-07-02 14:40:24.663 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-07-02T14:40:24.663474199Z 2026-07-02 14:40:24.663 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-07-02T14:40:24.727440674Z 2026-07-02 14:40:24.726 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-07-02T14:40:25.165989019Z 2026-07-02 14:40:25.165 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-07-02T14:40:25.188631696Z 2026-07-02 14:40:25.188 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:25.239904046Z 2026-07-02 14:40:25.239 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-07-02T14:40:25.268973442Z 2026-07-02 14:40:25.267 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:25.268996752Z 2026-07-02 14:40:25.268 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:25.391258457Z 2026-07-02 14:40:25.390 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-07-02T14:40:25.416567818Z 2026-07-02 14:40:25.416 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:25.427240372Z 2026-07-02 14:40:25.426 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-07-02T14:40:25.433138397Z 2026-07-02 14:40:25.432 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72059597335363584-solr-node1:8983_solr-n_0000000000) starting
2026-07-02T14:40:25.512875303Z 2026-07-02 14:40:25.512 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-07-02T14:40:25.515794951Z 2026-07-02 14:40:25.515 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-07-02T14:40:25.531799378Z 2026-07-02 14:40:25.531 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-07-02T14:40:25.625031015Z 2026-07-02 14:40:25.624 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-07-02T14:40:26.257483407Z 2026-07-02 14:40:26.256 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-07-02T14:40:26.533915670Z 2026-07-02 14:40:26.533 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-07-02T14:40:26.992184947Z 2026-07-02 14:40:26.991 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:27.603602322Z 2026-07-02 14:40:27.602 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-07-02T14:40:27.612007952Z 2026-07-02 14:40:27.611 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_07_02.request.log
2026-07-02T14:40:27.617091138Z 2026-07-02 14:40:27.616 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-07-02T14:40:27.618631115Z 2026-07-02 14:40:27.618 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @7983ms
2026-07-02T14:40:27.937396488Z 2026-07-02 14:40:27.936 INFO (qtp985324122-28-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=78
2026-07-02T14:40:28.208672071Z 2026-07-02 14:40:28.208 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=21
2026-07-02T14:40:33.073907368Z 2026-07-02 14:40:33.073 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-07-02T14:40:39.231537059Z 2026-07-02 14:40:39.230 INFO (qtp985324122-100-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=34
2026-07-02T14:40:39.267339298Z 2026-07-02 14:40:39.266 INFO (qtp985324122-100-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-07-02T14:40:39.303995831Z 2026-07-02 14:40:39.303 INFO (qtp985324122-28-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=4
2026-07-02T14:40:39.723619301Z 2026-07-02 14:40:39.723 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-07-02T14:40:39.763159329Z 2026-07-02 14:40:39.762 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-07-02T14:40:39.785445186Z 2026-07-02 14:40:39.784 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:39.935884771Z 2026-07-02 14:40:39.934 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-07-02T14:40:39.935909838Z "core":"test-collection_shard3_replica_n1",
2026-07-02T14:40:39.935913351Z "node_name":"solr-node1:8983_solr",
2026-07-02T14:40:39.935916690Z "base_url":"http://solr-node1:8983/solr",
2026-07-02T14:40:39.935919519Z "collection":"test-collection",
2026-07-02T14:40:39.935922345Z "shard":"shard3",
2026-07-02T14:40:39.935924902Z "state":"down",
2026-07-02T14:40:39.935927620Z "type":"NRT",
2026-07-02T14:40:39.935930128Z "operation":"ADDREPLICA",
2026-07-02T14:40:39.935932787Z "waitForFinalState":"false"}
2026-07-02T14:40:39.947205925Z 2026-07-02 14:40:39.946 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-07-02T14:40:39.947237095Z "core":"test-collection_shard2_replica_n2",
2026-07-02T14:40:39.947241815Z "node_name":"solr-node3:8985_solr",
2026-07-02T14:40:39.947245870Z "base_url":"http://solr-node3:8985/solr",
2026-07-02T14:40:39.947248765Z "collection":"test-collection",
2026-07-02T14:40:39.947251444Z "shard":"shard2",
2026-07-02T14:40:39.947254281Z "state":"down",
2026-07-02T14:40:39.947256996Z "type":"NRT",
2026-07-02T14:40:39.947259516Z "operation":"ADDREPLICA",
2026-07-02T14:40:39.947262022Z "waitForFinalState":"false"}
2026-07-02T14:40:39.962030489Z 2026-07-02 14:40:39.961 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-07-02T14:40:39.962052407Z "core":"test-collection_shard1_replica_n4",
2026-07-02T14:40:39.962056250Z "node_name":"solr-node2:8984_solr",
2026-07-02T14:40:39.962059210Z "base_url":"http://solr-node2:8984/solr",
2026-07-02T14:40:39.962061850Z "collection":"test-collection",
2026-07-02T14:40:39.962077134Z "shard":"shard1",
2026-07-02T14:40:39.962079437Z "state":"down",
2026-07-02T14:40:39.962081354Z "type":"NRT",
2026-07-02T14:40:39.962083368Z "operation":"ADDREPLICA",
2026-07-02T14:40:39.962085306Z "waitForFinalState":"false"}
2026-07-02T14:40:40.074363039Z 2026-07-02 14:40:40.073 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:40.214284766Z 2026-07-02 14:40:40.213 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-07-02T14:40:40.336736309Z 2026-07-02 14:40:40.336 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-07-02T14:40:40.423804240Z 2026-07-02 14:40:40.423 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-07-02T14:40:41.150804086Z 2026-07-02 14:40:41.150 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-07-02T14:40:41.150957942Z 2026-07-02 14:40:41.150 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-07-02T14:40:41.206620617Z 2026-07-02 14:40:41.206 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-07-02T14:40:42.525124465Z 2026-07-02 14:40:42.524 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:42.567053871Z 2026-07-02 14:40:42.566 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-07-02T14:40:42.570622160Z 2026-07-02 14:40:42.570 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-07-02T14:40:42.571922299Z 2026-07-02 14:40:42.571 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-07-02T14:40:42.572027684Z 2026-07-02 14:40:42.571 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-07-02T14:40:42.638759772Z 2026-07-02 14:40:42.638 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-07-02T14:40:42.642074881Z 2026-07-02 14:40:42.641 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-07-02T14:40:42.644548387Z 2026-07-02 14:40:42.644 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-07-02T14:40:42.646297236Z 2026-07-02 14:40:42.646 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-07-02T14:40:42.686270974Z 2026-07-02 14:40:42.685 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-07-02T14:40:42.694682220Z 2026-07-02 14:40:42.694 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-07-02T14:40:42.703452624Z 2026-07-02 14:40:42.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-07-02T14:40:42.703474824Z 2026-07-02 14:40:42.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-07-02T14:40:42.712820919Z 2026-07-02 14:40:42.712 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-07-02T14:40:42.722506150Z 2026-07-02 14:40:42.721 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-07-02T14:40:42.727160303Z 2026-07-02 14:40:42.726 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-07-02T14:40:42.742306775Z 2026-07-02 14:40:42.741 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-07-02T14:40:42.742328707Z 2026-07-02 14:40:42.741 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-07-02T14:40:42.742382374Z 2026-07-02 14:40:42.742 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-07-02T14:40:42.743890302Z 2026-07-02 14:40:42.743 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-07-02T14:40:42.744350903Z 2026-07-02 14:40:42.744 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-07-02T14:40:42.745526000Z 2026-07-02 14:40:42.745 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72059597335363584-core_node3-n_0000000000
2026-07-02T14:40:42.756704381Z 2026-07-02 14:40:42.756 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-07-02T14:40:42.864440217Z 2026-07-02 14:40:42.863 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.867782713Z 2026-07-02 14:40:42.867 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-07-02T14:40:42.877781749Z 2026-07-02 14:40:42.877 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2664
2026-07-02T14:40:42.975506932Z 2026-07-02 14:40:42.975 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1697989,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-07-02T14:40:18.042463931Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-07-02T14:40:28.873517158Z *** [WARN] *** Your open file limit is currently 1024.
2026-07-02T14:40:28.873550859Z It should be set to 65000 to avoid operational disruption.
2026-07-02T14:40:28.873565014Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-07-02T14:40:28.873806035Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-07-02T14:40:28.873817912Z
2026-07-02T14:40:28.883294564Z Java 17 detected. Enabled workaround for SOLR-16463
2026-07-02T14:40:28.893826930Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-07-02T14:40:28.954781053Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-07-02T14:40:28.988402216Z WARNING: A command line option has enabled the Security Manager
2026-07-02T14:40:28.988420870Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-07-02T14:40:30.528390223Z 2026-07-02 14:40:30.525 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-07-02T14:40:30.996244436Z 2026-07-02 14:40:30.995 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-07-02T14:40:31.041527575Z 2026-07-02 14:40:31.041 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-07-02T14:40:31.052516299Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-07-02T14:40:31.052529653Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-07-02T14:40:31.052532425Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-07-02T14:40:31.052534989Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-07-02T14:40:31.051974839Z
2026-07-02T14:40:31.055707618Z 2026-07-02 14:40:31.055 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-07-02T14:40:31.106433037Z 2026-07-02 14:40:31.105 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-07-02T14:40:31.157075522Z 2026-07-02 14:40:31.156 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.431156907Z 2026-07-02 14:40:31.430 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-07-02T14:40:31.454653697Z 2026-07-02 14:40:31.454 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:31.454734416Z 2026-07-02 14:40:31.454 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:31.454887540Z 2026-07-02 14:40:31.454 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.567865591Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-07-02T14:40:31.568143238Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-07-02T14:40:31.568247707Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-07-02T14:40:31.623879241Z 2026-07-02 14:40:31.623 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-07-02T14:40:32.670773520Z 2026-07-02 14:40:32.670 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-07-02T14:40:32.917103883Z 2026-07-02 14:40:32.916 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-07-02T14:40:32.918937607Z 2026-07-02 14:40:32.916 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-07-02T14:40:32.931284192Z 2026-07-02 14:40:32.930 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:32.938593211Z 2026-07-02 14:40:32.938 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-07-02T14:40:32.943340882Z 2026-07-02 14:40:32.942 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:32.943357336Z 2026-07-02 14:40:32.942 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:32.986631343Z 2026-07-02 14:40:32.986 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-07-02T14:40:32.996822064Z 2026-07-02 14:40:32.996 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-07-02T14:40:33.035650631Z 2026-07-02 14:40:33.035 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:33.052647661Z 2026-07-02 14:40:33.052 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-07-02T14:40:33.063711593Z 2026-07-02 14:40:33.063 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-07-02T14:40:33.075390380Z 2026-07-02 14:40:33.074 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-07-02T14:40:33.144925977Z 2026-07-02 14:40:33.144 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-07-02T14:40:33.513093466Z 2026-07-02 14:40:33.512 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-07-02T14:40:33.785251972Z 2026-07-02 14:40:33.784 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-07-02T14:40:34.079897622Z 2026-07-02 14:40:34.079 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:34.236641933Z 2026-07-02 14:40:34.235 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-07-02T14:40:34.261383026Z 2026-07-02 14:40:34.260 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_07_02.request.log
2026-07-02T14:40:34.265992866Z 2026-07-02 14:40:34.265 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-07-02T14:40:34.267454138Z 2026-07-02 14:40:34.267 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5375ms
2026-07-02T14:40:34.518090223Z 2026-07-02 14:40:34.517 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=63
2026-07-02T14:40:34.592513171Z 2026-07-02 14:40:34.592 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-07-02T14:40:40.213268726Z 2026-07-02 14:40:40.212 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-07-02T14:40:40.276056868Z 2026-07-02 14:40:40.275 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-07-02T14:40:40.380750327Z 2026-07-02 14:40:40.380 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-07-02T14:40:40.874932471Z 2026-07-02 14:40:40.874 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-07-02T14:40:40.875009977Z 2026-07-02 14:40:40.874 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-07-02T14:40:40.925174604Z 2026-07-02 14:40:40.924 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-07-02T14:40:41.530252064Z 2026-07-02 14:40:41.529 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:41.566992782Z 2026-07-02 14:40:41.566 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-07-02T14:40:41.581045691Z 2026-07-02 14:40:41.580 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-07-02T14:40:41.582484694Z 2026-07-02 14:40:41.581 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-07-02T14:40:41.582571129Z 2026-07-02 14:40:41.582 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-07-02T14:40:41.639466313Z 2026-07-02 14:40:41.639 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-07-02T14:40:41.642456580Z 2026-07-02 14:40:41.642 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-07-02T14:40:41.645148732Z 2026-07-02 14:40:41.644 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-07-02T14:40:41.647060708Z 2026-07-02 14:40:41.646 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-07-02T14:40:41.666505561Z 2026-07-02 14:40:41.665 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-07-02T14:40:41.674128927Z 2026-07-02 14:40:41.673 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-07-02T14:40:41.681471660Z 2026-07-02 14:40:41.680 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-07-02T14:40:41.681478974Z 2026-07-02 14:40:41.681 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-07-02T14:40:41.688267505Z 2026-07-02 14:40:41.687 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-07-02T14:40:41.702466414Z 2026-07-02 14:40:41.701 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-07-02T14:40:41.704028873Z 2026-07-02 14:40:41.703 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-07-02T14:40:41.721438020Z 2026-07-02 14:40:41.720 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-07-02T14:40:41.721444155Z 2026-07-02 14:40:41.721 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-07-02T14:40:41.721502134Z 2026-07-02 14:40:41.721 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-07-02T14:40:41.723525605Z 2026-07-02 14:40:41.723 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-07-02T14:40:41.724497822Z 2026-07-02 14:40:41.724 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-07-02T14:40:41.725848862Z 2026-07-02 14:40:41.725 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72059597335363588-core_node6-n_0000000000
2026-07-02T14:40:41.739387645Z 2026-07-02 14:40:41.738 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-07-02T14:40:41.850247969Z 2026-07-02 14:40:41.849 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:41.856462041Z 2026-07-02 14:40:41.856 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-07-02T14:40:41.866804119Z 2026-07-02 14:40:41.866 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1655
2026-07-02T14:40:41.972863700Z 2026-07-02 14:40:41.972 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.864794513Z 2026-07-02 14:40:42.864 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.975789809Z 2026-07-02 14:40:42.975 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1698390,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-07-02T14:40:28.467845525Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-07-02T14:40:28.881193194Z *** [WARN] *** Your open file limit is currently 1024.
2026-07-02T14:40:28.881221498Z It should be set to 65000 to avoid operational disruption.
2026-07-02T14:40:28.881224568Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-07-02T14:40:28.881550800Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-07-02T14:40:28.881600193Z
2026-07-02T14:40:28.890985463Z Java 17 detected. Enabled workaround for SOLR-16463
2026-07-02T14:40:28.898465185Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-07-02T14:40:28.951650107Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-07-02T14:40:28.988393781Z WARNING: A command line option has enabled the Security Manager
2026-07-02T14:40:28.988416004Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-07-02T14:40:30.528346910Z 2026-07-02 14:40:30.525 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-07-02T14:40:30.996450459Z 2026-07-02 14:40:30.995 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-07-02T14:40:31.041659604Z 2026-07-02 14:40:31.041 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-07-02T14:40:31.052040007Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-07-02T14:40:31.052068071Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-07-02T14:40:31.052232491Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-07-02T14:40:31.052351865Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-07-02T14:40:31.051794634Z
2026-07-02T14:40:31.055774586Z 2026-07-02 14:40:31.055 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-07-02T14:40:31.106478211Z 2026-07-02 14:40:31.105 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-07-02T14:40:31.157117541Z 2026-07-02 14:40:31.156 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.431216133Z 2026-07-02 14:40:31.430 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-07-02T14:40:31.455186365Z 2026-07-02 14:40:31.454 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:31.455298821Z 2026-07-02 14:40:31.454 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:31.455409860Z 2026-07-02 14:40:31.454 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.567909016Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-07-02T14:40:31.568165651Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-07-02T14:40:31.568258191Z 2026-07-02 14:40:31.568 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-07-02T14:40:31.623844801Z 2026-07-02 14:40:31.623 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-07-02T14:40:32.697848437Z 2026-07-02 14:40:32.697 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-07-02T14:40:32.915295940Z 2026-07-02 14:40:32.914 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-07-02T14:40:32.917503700Z 2026-07-02 14:40:32.914 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-07-02T14:40:32.930843640Z 2026-07-02 14:40:32.930 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:32.937833619Z 2026-07-02 14:40:32.937 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-07-02T14:40:32.942358012Z 2026-07-02 14:40:32.941 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:32.942735450Z 2026-07-02 14:40:32.942 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:32.986591929Z 2026-07-02 14:40:32.986 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-07-02T14:40:32.996738891Z 2026-07-02 14:40:32.996 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-07-02T14:40:33.035623932Z 2026-07-02 14:40:33.034 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:33.051132226Z 2026-07-02 14:40:33.050 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-07-02T14:40:33.063028215Z 2026-07-02 14:40:33.062 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-07-02T14:40:33.074869555Z 2026-07-02 14:40:33.074 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-07-02T14:40:33.144870702Z 2026-07-02 14:40:33.144 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-07-02T14:40:33.512665504Z 2026-07-02 14:40:33.512 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-07-02T14:40:33.786030358Z 2026-07-02 14:40:33.785 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-07-02T14:40:34.079862135Z 2026-07-02 14:40:34.079 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:34.236638173Z 2026-07-02 14:40:34.235 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-07-02T14:40:34.261397120Z 2026-07-02 14:40:34.260 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_07_02.request.log
2026-07-02T14:40:34.265963788Z 2026-07-02 14:40:34.265 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-07-02T14:40:34.267476534Z 2026-07-02 14:40:34.267 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5370ms
2026-07-02T14:40:37.251924595Z 2026-07-02 14:40:37.243 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=99
2026-07-02T14:40:37.344062894Z 2026-07-02 14:40:37.343 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=20
2026-07-02T14:40:39.677471980Z 2026-07-02 14:40:39.677 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=12
2026-07-02T14:40:40.214061264Z 2026-07-02 14:40:40.212 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-07-02T14:40:40.281009350Z 2026-07-02 14:40:40.280 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-07-02T14:40:40.374284331Z 2026-07-02 14:40:40.373 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-07-02T14:40:40.877444444Z 2026-07-02 14:40:40.876 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-07-02T14:40:40.877472738Z 2026-07-02 14:40:40.877 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-07-02T14:40:40.925401098Z 2026-07-02 14:40:40.924 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-07-02T14:40:41.536689951Z 2026-07-02 14:40:41.535 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:41.566829341Z 2026-07-02 14:40:41.566 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-07-02T14:40:41.581008826Z 2026-07-02 14:40:41.580 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-07-02T14:40:41.582448423Z 2026-07-02 14:40:41.581 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-07-02T14:40:41.582574912Z 2026-07-02 14:40:41.582 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-07-02T14:40:41.641692752Z 2026-07-02 14:40:41.641 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-07-02T14:40:41.645153260Z 2026-07-02 14:40:41.644 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-07-02T14:40:41.648061587Z 2026-07-02 14:40:41.647 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-07-02T14:40:41.650192718Z 2026-07-02 14:40:41.649 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-07-02T14:40:41.666473857Z 2026-07-02 14:40:41.666 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-07-02T14:40:41.674371488Z 2026-07-02 14:40:41.673 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-07-02T14:40:41.681440738Z 2026-07-02 14:40:41.681 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-07-02T14:40:41.681684643Z 2026-07-02 14:40:41.681 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-07-02T14:40:41.689183993Z 2026-07-02 14:40:41.688 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-07-02T14:40:41.702531091Z 2026-07-02 14:40:41.701 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-07-02T14:40:41.704317786Z 2026-07-02 14:40:41.703 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-07-02T14:40:41.721402860Z 2026-07-02 14:40:41.720 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-07-02T14:40:41.721711512Z 2026-07-02 14:40:41.721 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-07-02T14:40:41.721755382Z 2026-07-02 14:40:41.721 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-07-02T14:40:41.723499918Z 2026-07-02 14:40:41.723 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-07-02T14:40:41.724549549Z 2026-07-02 14:40:41.724 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-07-02T14:40:41.725942042Z 2026-07-02 14:40:41.725 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72059597335363587-core_node5-n_0000000000
2026-07-02T14:40:41.740774482Z 2026-07-02 14:40:41.740 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-07-02T14:40:41.850245383Z 2026-07-02 14:40:41.849 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:41.855136849Z 2026-07-02 14:40:41.854 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-07-02T14:40:41.866435220Z 2026-07-02 14:40:41.866 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1654
2026-07-02T14:40:41.973425225Z 2026-07-02 14:40:41.972 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.864829959Z 2026-07-02 14:40:42.864 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.891961804Z 2026-07-02 14:40:42.891 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-07-02T14:40:42.975872729Z 2026-07-02 14:40:42.975 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.978816889Z 2026-07-02 14:40:42.978 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3284
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1698520,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-07-02T14:40:28.67011128Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 81 Docker SolrCloud rolling upgrade # in 28729 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Image is up to date for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 65f0f8844d165941c9ddfc17dd49077bc1d625280a6e84478a8808dc70036917
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# ac6dc439c8913c4207abda402fcf40b0552514c8b66f9032b6952414f369b8ae
# Starting solr-node2 connected to first node's ZooKeeper
# 24b8a184b892a12fbcac81a86d1bf3be9113b0511cf7235f67fb0df817e68b6b
# Started solr-node3 connected to first node's ZooKeeper
# ee1ab1fdb8c719d1472210dfc2b7e96983adf269da059f60c063dfc98f496fdd
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-07-02 14:40:39.360; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:40:39.427; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 82 allowPaths - backup # in 16808 ms
ok 83 snapshot lifecycle # in 5123 ms
ok 84 snapshot list # in 3416 ms
ok 85 snapshot list via --solr-connection # in 3375 ms
ok 86 snapshot describe # in 3345 ms
ok 87 snapshot describe via --solr-connection # in 3370 ms
ok 88 start solr with ssl # in 16374 ms
ok 89 use different hostname when not checking peer-name # in 25064 ms
ok 90 start solr with ssl and auth # in 14992 ms
ok 91 start solr with client truststore and security manager # in 16817 ms
ok 92 start solr with mTLS needed # in 46371 ms
ok 93 start solr with mTLS wanted # in 34701 ms
not ok 94 test keystore reload # in 33351 ms
# (in test file test/test_ssl.bats, line 576)
# `wait_for 30 1 solr api --solr-url "https://localhost:${SOLR_PORT}/solr/admin/info/system"' failed with status 127
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Waiting up to 180 seconds to see Solr running on port 44869 [|] [/] [-] [\] [|]
# Started Solr server on port 44869 (pid=1713085). Happy searching!
#
# INFO - 2026-07-02 14:44:09.082; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-07-02 14:44:09.085; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-07-02 14:44:09.098; org.apache.solr.common.util.EnvUtils; Deprecated system property host has been replaced by solr.host.advertise. Support for the old property will be removed in a future version of Solr.
# INFO - 2026-07-02 14:44:09.374; org.eclipse.jetty.util.ssl.SslContextFactory; x509=X509@294a6b8e(cert1,h=[localhost],a=[/127.0.0.1],w=[]) for Client@4b1d6571[provider=null,keyStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server1.keystore.p12,trustStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server1.keystore.p12]
# Waiting up to 180 seconds to see Solr running on port 44870 [|] [/] [-] [\] [|]
# Started Solr server on port 44870 (pid=1713396). Happy searching!
#
# INFO - 2026-07-02 14:44:15.157; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-07-02 14:44:15.160; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-07-02 14:44:15.172; org.apache.solr.common.util.EnvUtils; Deprecated system property host has been replaced by solr.host.advertise. Support for the old property will be removed in a future version of Solr.
# INFO - 2026-07-02 14:44:15.452; org.eclipse.jetty.util.ssl.SslContextFactory; x509=X509@294a6b8e(cert1,h=[localhost],a=[/127.0.0.1],w=[]) for Client@4b1d6571[provider=null,keyStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server2.keystore.p12,trustStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server2.keystore.p12]
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_ssl.bats: line 576: wait_for: command not found
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/94-7
# Last output:
# INFO - 2026-07-02 14:44:31.876; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-07-02 14:44:31.879; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-07-02 14:44:31.892; org.apache.solr.common.util.EnvUtils; Deprecated system property host has been replaced by solr.host.advertise. Support for the old property will be removed in a future version of Solr.
# Neither --solr-connection, --zk-host or --solr-url parameters, nor SOLR_CONNECTION, ZK_HOST env var provided, so assuming solr url is https://localhost:44869.
# WARN - 2026-07-02 14:44:32.843; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:44:32.843; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 95 SOLR-11740 check 'solr stop' connection # in 15991 ms
not ok 96 stop command for single port # in 57579 ms
# (in test file test/test_start_solr.bats, line 48)
# `solr assert --started http://localhost:${SOLR2_PORT} --timeout 5000' failed with status 100
# Waiting up to 180 seconds to see Solr running on port 44869 [|] [/] [-] [\] [|]
# Started Solr server on port 44869 (pid=1715746). Happy searching!
#
# Waiting up to 180 seconds to see Solr running on port 44870 [|] [/] [-] [\] [|]
# Started Solr server on port 44870 (pid=1715880). Happy searching!
#
#
# ERROR: Solr is not running on url http://localhost:44870 after 5 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/96-2
ok 97 check stop command doesn't hang # in 221 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8471 ms
ok 99 deprecated system properties converted to modern properties # in 43818 ms
ok 100 start with custom jetty options # in 8212 ms
ok 101 -c flag prints no-op warning and still starts in cloud mode # in 9574 ms
ok 102 bootstrapping a configset # in 9520 ms
ok 103 status detects locally running solr # in 3014 ms
ok 104 status with --solr-url from user # in 1381 ms
ok 105 status with --port from user # in 1365 ms
ok 106 multiple connection options are prevented # in 816 ms
ok 107 status with invalid --solr-url from user # in 1142 ms
ok 108 status with --short format # in 767 ms
ok 109 searching solr via locally executed streaming expression # in 1786 ms
ok 110 searching solr via remotely executed streaming expression by solr-url # in 1572 ms
ok 111 searching solr via remotely executed streaming expression by solr connection # in 1255 ms
ok 112 variable interpolation # in 1684 ms
ok 113 searching solr without credentials fails with error # in 1154 ms
ok 114 --version returns Solr version # in 747 ms
ok 115 version as direct tool call still runs # in 766 ms
ok 116 short help # in 893 ms
ok 117 short help is inferred # in 753 ms
ok 118 long help # in 794 ms
ok 119 running subcommands with zk is prevented # in 773 ms
ok 120 listing out files # in 1168 ms
ok 121 connecting to solr via various solr urls and zk hosts # in 8405 ms
ok 122 copying files around # in 8185 ms
not ok 123 upconfig # in 1478 ms
# (in test file test/test_zk.bats, line 123)
# `wait_for 10 1 bash -c "curl -s 'http://localhost:${SOLR_PORT}/api/configsets' | grep -q techproducts2"' failed with status 127
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_zk.bats: line 123: wait_for: command not found
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/123-8
# Last output:
# WARN - 2026-07-02 14:48:09.702; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:48:09.706; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Uploading /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server/solr/configsets/sample_techproducts_configs/conf for config techproducts2 to ZooKeeper at localhost:45869
ok 124 SOLR-12429 test upconfig fails with symlink # in 1193 ms
ok 125 downconfig # in 1208 ms
ok 126 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1461 ms
ok 127 env var ZK_HOST is honored # in 2143 ms
ok 128 Cleanup # in 1917 ms
> Task :solr:packaging:integrationTests FAILED
[Incubating] Problems report is available at: file:///home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 354
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests' (registered in build file 'solr/packaging/build.gradle').
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/9.6.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 22m 40s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/h67zf5wfrehxg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 250 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/250/
No tests ran.
Build Log:
[...truncated 969 lines...]
BUILD FAILED in 29m 30s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vg4q4fekx2ryy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any