You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@solr.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2026/03/04 13:29:22 UTC
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 138 - Still Failing!
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/138/
No tests ran.
Build Log:
[...truncated 900 lines...]
BUILD FAILED in 29m 57s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/qvbbkbuhimlag
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 251 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/251/
No tests ran.
Build Log:
[...truncated 350 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-07-02 14:31:15.001; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:15.004; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:15.472; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:15.472; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1314 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 44869 is already being used by another process (pid: 1668090)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-07-02 14:31:16.641; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:31:16.644; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 17570 ms
ok 11 package with basic auth # in 12254 ms
ok 12 admin UI static content is excepted from basic auth # in 10586 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 144 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 81 ms
ok 16 compressed response can be decompressed and parsed # in 78 ms
ok 17 setting property # in 10995 ms
ok 18 short form of setting property # in 6461 ms
ok 19 connecting to solr via various connection types # in 10503 ms
ok 20 create for non cloud mode # in 8075 ms
ok 21 create for cloud mode # in 10187 ms
ok 22 multiple connection options are prevented # in 869 ms
ok 23 create collection # in 7815 ms
ok 24 create collection using solrUrl # in 5242 ms
ok 25 create collection using legacy solrUrl # in 5361 ms
ok 26 create collection using Zookeeper # in 5508 ms
ok 27 reject d option with invalid config dir # in 2714 ms
ok 28 accept d option with builtin config # in 5322 ms
ok 29 accept d option with explicit path to config # in 5285 ms
ok 30 accept n option as config name # in 5286 ms
ok 31 allow config reuse when n option specifies same config # in 9163 ms
ok 32 create multisharded collections when s provided # in 5321 ms
ok 33 create replicated collections when rf provided # in 6149 ms
ok 34 can delete collections # in 10977 ms
ok 35 can delete collections with solr-url # in 8562 ms
ok 36 can delete collections with solr-connection # in 8062 ms
ok 37 collection delete also deletes zk config # in 7409 ms
ok 38 deletes accompanying zk config with nondefault name # in 7317 ms
ok 39 delete-config option can opt to leave config in zk # in 7150 ms
ok 40 start -e cloud works with --no-prompt # in 2705 ms
ok 41 start -e cloud works with --jvm-opts # in 106 ms
ok 42 Check export command # in 18993 ms
ok 43 export fails on non cloud mode # in 9613 ms
Waiting for Tika Server to be ready on port 44874
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16574 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16538 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16543 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 13349 ms
ok 48 healthcheck errors on standalone solr # in 6811 ms
ok 49 solr help flag prints help # in 752 ms
ok 50 solr with no flags prints help # in 740 ms
ok 51 start help flag prints help # in 147 ms
ok 52 start h flag prints help # in 143 ms
ok 53 stop help flag prints help # in 148 ms
ok 54 restart help flag prints help # in 144 ms
ok 55 status help flag prints help # in 788 ms
ok 56 healthcheck help flag prints help # in 753 ms
ok 57 create help flag prints help # in 751 ms
ok 58 delete help flag prints help # in 812 ms
ok 59 zk help flag prints help # in 766 ms
ok 60 auth help flag prints help # in 761 ms
ok 61 assert help flag prints help # in 754 ms
ok 62 post help flag prints help # in 779 ms
ok 63 SQL Module # in 16272 ms
ok 64 icu collation in analysis-extras module # in 12534 ms
ok 65 Check lifecycle of sentiment classification # in 23561 ms
ok 66 Affinity placement plugin using sysprop # in 15839 ms
ok 67 Random placement plugin using ENV # in 15305 ms
ok 68 Check help command # in 2316 ms
ok 69 basic post with a type specified # in 5487 ms
ok 70 basic post with collection instead of url parameter # in 3964 ms
ok 71 basic post with solr-url and collection # in 3962 ms
ok 72 basic post with -s (solr-connection short form) and collection # in 3932 ms
ok 73 basic post WITHOUT a type specified # in 11294 ms
ok 74 crawling a directory as a dry-run # in 763 ms
ok 75 crawling a directory # in 4294 ms
ok 76 skipcommit and optimize and delete # in 3784 ms
ok 77 args mode # in 6848 ms
ok 78 stdin mode # in 3407 ms
ok 79 verbose echo the Solr response # in 3694 ms
ok 80 post solr log into solr via cli # in 9744 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 25 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-07-02T14:40:19.606527161Z *** [WARN] *** Your open file limit is currently 1024.
2026-07-02T14:40:19.606572796Z It should be set to 65000 to avoid operational disruption.
2026-07-02T14:40:19.606577499Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-07-02T14:40:19.606830320Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-07-02T14:40:19.606833549Z
2026-07-02T14:40:19.623828557Z Java 17 detected. Enabled workaround for SOLR-16463
2026-07-02T14:40:19.636344550Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-07-02T14:40:19.692223525Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-07-02T14:40:19.739373337Z WARNING: A command line option has enabled the Security Manager
2026-07-02T14:40:19.739403993Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-07-02T14:40:21.923545647Z 2026-07-02 14:40:21.918 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-07-02T14:40:22.459142044Z 2026-07-02 14:40:22.458 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-07-02T14:40:22.511966025Z 2026-07-02 14:40:22.511 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-07-02T14:40:22.525781070Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-07-02T14:40:22.525850140Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-07-02T14:40:22.526079965Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-07-02T14:40:22.526086653Z 2026-07-02 14:40:22.525 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-07-02T14:40:22.525548069Z
2026-07-02T14:40:22.529902728Z 2026-07-02 14:40:22.529 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-07-02T14:40:22.575286985Z 2026-07-02 14:40:22.574 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-07-02T14:40:22.624536554Z 2026-07-02 14:40:22.624 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-07-02T14:40:22.624652769Z 2026-07-02 14:40:22.624 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-07-02T14:40:22.714572874Z 2026-07-02 14:40:22.713 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-07-02T14:40:24.178740445Z 2026-07-02 14:40:24.178 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-07-02T14:40:24.645719265Z 2026-07-02 14:40:24.645 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-07-02T14:40:24.658455148Z 2026-07-02 14:40:24.657 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-07-02T14:40:24.658478216Z 2026-07-02 14:40:24.657 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-07-02T14:40:24.663425254Z 2026-07-02 14:40:24.663 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-07-02T14:40:24.663474199Z 2026-07-02 14:40:24.663 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-07-02T14:40:24.727440674Z 2026-07-02 14:40:24.726 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-07-02T14:40:25.165989019Z 2026-07-02 14:40:25.165 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-07-02T14:40:25.188631696Z 2026-07-02 14:40:25.188 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:25.239904046Z 2026-07-02 14:40:25.239 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-07-02T14:40:25.268973442Z 2026-07-02 14:40:25.267 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:25.268996752Z 2026-07-02 14:40:25.268 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:25.391258457Z 2026-07-02 14:40:25.390 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-07-02T14:40:25.416567818Z 2026-07-02 14:40:25.416 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:25.427240372Z 2026-07-02 14:40:25.426 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-07-02T14:40:25.433138397Z 2026-07-02 14:40:25.432 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72059597335363584-solr-node1:8983_solr-n_0000000000) starting
2026-07-02T14:40:25.512875303Z 2026-07-02 14:40:25.512 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-07-02T14:40:25.515794951Z 2026-07-02 14:40:25.515 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-07-02T14:40:25.531799378Z 2026-07-02 14:40:25.531 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-07-02T14:40:25.625031015Z 2026-07-02 14:40:25.624 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-07-02T14:40:26.257483407Z 2026-07-02 14:40:26.256 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-07-02T14:40:26.533915670Z 2026-07-02 14:40:26.533 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-07-02T14:40:26.992184947Z 2026-07-02 14:40:26.991 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:27.603602322Z 2026-07-02 14:40:27.602 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-07-02T14:40:27.612007952Z 2026-07-02 14:40:27.611 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_07_02.request.log
2026-07-02T14:40:27.617091138Z 2026-07-02 14:40:27.616 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-07-02T14:40:27.618631115Z 2026-07-02 14:40:27.618 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @7983ms
2026-07-02T14:40:27.937396488Z 2026-07-02 14:40:27.936 INFO (qtp985324122-28-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=78
2026-07-02T14:40:28.208672071Z 2026-07-02 14:40:28.208 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=21
2026-07-02T14:40:33.073907368Z 2026-07-02 14:40:33.073 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-07-02T14:40:39.231537059Z 2026-07-02 14:40:39.230 INFO (qtp985324122-100-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=34
2026-07-02T14:40:39.267339298Z 2026-07-02 14:40:39.266 INFO (qtp985324122-100-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-07-02T14:40:39.303995831Z 2026-07-02 14:40:39.303 INFO (qtp985324122-28-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=4
2026-07-02T14:40:39.723619301Z 2026-07-02 14:40:39.723 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-07-02T14:40:39.763159329Z 2026-07-02 14:40:39.762 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-07-02T14:40:39.785445186Z 2026-07-02 14:40:39.784 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:39.935884771Z 2026-07-02 14:40:39.934 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-07-02T14:40:39.935909838Z "core":"test-collection_shard3_replica_n1",
2026-07-02T14:40:39.935913351Z "node_name":"solr-node1:8983_solr",
2026-07-02T14:40:39.935916690Z "base_url":"http://solr-node1:8983/solr",
2026-07-02T14:40:39.935919519Z "collection":"test-collection",
2026-07-02T14:40:39.935922345Z "shard":"shard3",
2026-07-02T14:40:39.935924902Z "state":"down",
2026-07-02T14:40:39.935927620Z "type":"NRT",
2026-07-02T14:40:39.935930128Z "operation":"ADDREPLICA",
2026-07-02T14:40:39.935932787Z "waitForFinalState":"false"}
2026-07-02T14:40:39.947205925Z 2026-07-02 14:40:39.946 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-07-02T14:40:39.947237095Z "core":"test-collection_shard2_replica_n2",
2026-07-02T14:40:39.947241815Z "node_name":"solr-node3:8985_solr",
2026-07-02T14:40:39.947245870Z "base_url":"http://solr-node3:8985/solr",
2026-07-02T14:40:39.947248765Z "collection":"test-collection",
2026-07-02T14:40:39.947251444Z "shard":"shard2",
2026-07-02T14:40:39.947254281Z "state":"down",
2026-07-02T14:40:39.947256996Z "type":"NRT",
2026-07-02T14:40:39.947259516Z "operation":"ADDREPLICA",
2026-07-02T14:40:39.947262022Z "waitForFinalState":"false"}
2026-07-02T14:40:39.962030489Z 2026-07-02 14:40:39.961 INFO (OverseerStateUpdate-72059597335363584-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-07-02T14:40:39.962052407Z "core":"test-collection_shard1_replica_n4",
2026-07-02T14:40:39.962056250Z "node_name":"solr-node2:8984_solr",
2026-07-02T14:40:39.962059210Z "base_url":"http://solr-node2:8984/solr",
2026-07-02T14:40:39.962061850Z "collection":"test-collection",
2026-07-02T14:40:39.962077134Z "shard":"shard1",
2026-07-02T14:40:39.962079437Z "state":"down",
2026-07-02T14:40:39.962081354Z "type":"NRT",
2026-07-02T14:40:39.962083368Z "operation":"ADDREPLICA",
2026-07-02T14:40:39.962085306Z "waitForFinalState":"false"}
2026-07-02T14:40:40.074363039Z 2026-07-02 14:40:40.073 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:40.214284766Z 2026-07-02 14:40:40.213 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-07-02T14:40:40.336736309Z 2026-07-02 14:40:40.336 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-07-02T14:40:40.423804240Z 2026-07-02 14:40:40.423 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-07-02T14:40:41.150804086Z 2026-07-02 14:40:41.150 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-07-02T14:40:41.150957942Z 2026-07-02 14:40:41.150 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-07-02T14:40:41.206620617Z 2026-07-02 14:40:41.206 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-07-02T14:40:42.525124465Z 2026-07-02 14:40:42.524 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:42.567053871Z 2026-07-02 14:40:42.566 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-07-02T14:40:42.570622160Z 2026-07-02 14:40:42.570 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-07-02T14:40:42.571922299Z 2026-07-02 14:40:42.571 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-07-02T14:40:42.572027684Z 2026-07-02 14:40:42.571 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-07-02T14:40:42.638759772Z 2026-07-02 14:40:42.638 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-07-02T14:40:42.642074881Z 2026-07-02 14:40:42.641 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-07-02T14:40:42.644548387Z 2026-07-02 14:40:42.644 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-07-02T14:40:42.646297236Z 2026-07-02 14:40:42.646 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-07-02T14:40:42.686270974Z 2026-07-02 14:40:42.685 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-07-02T14:40:42.694682220Z 2026-07-02 14:40:42.694 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-07-02T14:40:42.703452624Z 2026-07-02 14:40:42.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-07-02T14:40:42.703474824Z 2026-07-02 14:40:42.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-07-02T14:40:42.712820919Z 2026-07-02 14:40:42.712 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-07-02T14:40:42.722506150Z 2026-07-02 14:40:42.721 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-07-02T14:40:42.727160303Z 2026-07-02 14:40:42.726 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-07-02T14:40:42.742306775Z 2026-07-02 14:40:42.741 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-07-02T14:40:42.742328707Z 2026-07-02 14:40:42.741 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-07-02T14:40:42.742382374Z 2026-07-02 14:40:42.742 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-07-02T14:40:42.743890302Z 2026-07-02 14:40:42.743 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-07-02T14:40:42.744350903Z 2026-07-02 14:40:42.744 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-07-02T14:40:42.745526000Z 2026-07-02 14:40:42.745 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72059597335363584-core_node3-n_0000000000
2026-07-02T14:40:42.756704381Z 2026-07-02 14:40:42.756 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-07-02T14:40:42.864440217Z 2026-07-02 14:40:42.863 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.867782713Z 2026-07-02 14:40:42.867 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-07-02T14:40:42.877781749Z 2026-07-02 14:40:42.877 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2664
2026-07-02T14:40:42.975506932Z 2026-07-02 14:40:42.975 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1697989,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-07-02T14:40:18.042463931Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-07-02T14:40:28.873517158Z *** [WARN] *** Your open file limit is currently 1024.
2026-07-02T14:40:28.873550859Z It should be set to 65000 to avoid operational disruption.
2026-07-02T14:40:28.873565014Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-07-02T14:40:28.873806035Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-07-02T14:40:28.873817912Z
2026-07-02T14:40:28.883294564Z Java 17 detected. Enabled workaround for SOLR-16463
2026-07-02T14:40:28.893826930Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-07-02T14:40:28.954781053Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-07-02T14:40:28.988402216Z WARNING: A command line option has enabled the Security Manager
2026-07-02T14:40:28.988420870Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-07-02T14:40:30.528390223Z 2026-07-02 14:40:30.525 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-07-02T14:40:30.996244436Z 2026-07-02 14:40:30.995 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-07-02T14:40:31.041527575Z 2026-07-02 14:40:31.041 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-07-02T14:40:31.052516299Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-07-02T14:40:31.052529653Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-07-02T14:40:31.052532425Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-07-02T14:40:31.052534989Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-07-02T14:40:31.051974839Z
2026-07-02T14:40:31.055707618Z 2026-07-02 14:40:31.055 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-07-02T14:40:31.106433037Z 2026-07-02 14:40:31.105 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-07-02T14:40:31.157075522Z 2026-07-02 14:40:31.156 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.431156907Z 2026-07-02 14:40:31.430 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-07-02T14:40:31.454653697Z 2026-07-02 14:40:31.454 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:31.454734416Z 2026-07-02 14:40:31.454 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:31.454887540Z 2026-07-02 14:40:31.454 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.567865591Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-07-02T14:40:31.568143238Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-07-02T14:40:31.568247707Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-07-02T14:40:31.623879241Z 2026-07-02 14:40:31.623 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-07-02T14:40:32.670773520Z 2026-07-02 14:40:32.670 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-07-02T14:40:32.917103883Z 2026-07-02 14:40:32.916 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-07-02T14:40:32.918937607Z 2026-07-02 14:40:32.916 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-07-02T14:40:32.931284192Z 2026-07-02 14:40:32.930 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:32.938593211Z 2026-07-02 14:40:32.938 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-07-02T14:40:32.943340882Z 2026-07-02 14:40:32.942 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:32.943357336Z 2026-07-02 14:40:32.942 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:32.986631343Z 2026-07-02 14:40:32.986 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-07-02T14:40:32.996822064Z 2026-07-02 14:40:32.996 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-07-02T14:40:33.035650631Z 2026-07-02 14:40:33.035 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:33.052647661Z 2026-07-02 14:40:33.052 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-07-02T14:40:33.063711593Z 2026-07-02 14:40:33.063 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-07-02T14:40:33.075390380Z 2026-07-02 14:40:33.074 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-07-02T14:40:33.144925977Z 2026-07-02 14:40:33.144 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-07-02T14:40:33.513093466Z 2026-07-02 14:40:33.512 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-07-02T14:40:33.785251972Z 2026-07-02 14:40:33.784 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-07-02T14:40:34.079897622Z 2026-07-02 14:40:34.079 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:34.236641933Z 2026-07-02 14:40:34.235 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-07-02T14:40:34.261383026Z 2026-07-02 14:40:34.260 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_07_02.request.log
2026-07-02T14:40:34.265992866Z 2026-07-02 14:40:34.265 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-07-02T14:40:34.267454138Z 2026-07-02 14:40:34.267 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5375ms
2026-07-02T14:40:34.518090223Z 2026-07-02 14:40:34.517 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=63
2026-07-02T14:40:34.592513171Z 2026-07-02 14:40:34.592 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-07-02T14:40:40.213268726Z 2026-07-02 14:40:40.212 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-07-02T14:40:40.276056868Z 2026-07-02 14:40:40.275 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-07-02T14:40:40.380750327Z 2026-07-02 14:40:40.380 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-07-02T14:40:40.874932471Z 2026-07-02 14:40:40.874 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-07-02T14:40:40.875009977Z 2026-07-02 14:40:40.874 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-07-02T14:40:40.925174604Z 2026-07-02 14:40:40.924 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-07-02T14:40:41.530252064Z 2026-07-02 14:40:41.529 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:41.566992782Z 2026-07-02 14:40:41.566 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-07-02T14:40:41.581045691Z 2026-07-02 14:40:41.580 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-07-02T14:40:41.582484694Z 2026-07-02 14:40:41.581 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-07-02T14:40:41.582571129Z 2026-07-02 14:40:41.582 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-07-02T14:40:41.639466313Z 2026-07-02 14:40:41.639 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-07-02T14:40:41.642456580Z 2026-07-02 14:40:41.642 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-07-02T14:40:41.645148732Z 2026-07-02 14:40:41.644 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-07-02T14:40:41.647060708Z 2026-07-02 14:40:41.646 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-07-02T14:40:41.666505561Z 2026-07-02 14:40:41.665 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-07-02T14:40:41.674128927Z 2026-07-02 14:40:41.673 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-07-02T14:40:41.681471660Z 2026-07-02 14:40:41.680 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-07-02T14:40:41.681478974Z 2026-07-02 14:40:41.681 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-07-02T14:40:41.688267505Z 2026-07-02 14:40:41.687 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-07-02T14:40:41.702466414Z 2026-07-02 14:40:41.701 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-07-02T14:40:41.704028873Z 2026-07-02 14:40:41.703 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-07-02T14:40:41.721438020Z 2026-07-02 14:40:41.720 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-07-02T14:40:41.721444155Z 2026-07-02 14:40:41.721 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-07-02T14:40:41.721502134Z 2026-07-02 14:40:41.721 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-07-02T14:40:41.723525605Z 2026-07-02 14:40:41.723 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-07-02T14:40:41.724497822Z 2026-07-02 14:40:41.724 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-07-02T14:40:41.725848862Z 2026-07-02 14:40:41.725 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72059597335363588-core_node6-n_0000000000
2026-07-02T14:40:41.739387645Z 2026-07-02 14:40:41.738 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-07-02T14:40:41.850247969Z 2026-07-02 14:40:41.849 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:41.856462041Z 2026-07-02 14:40:41.856 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-07-02T14:40:41.866804119Z 2026-07-02 14:40:41.866 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1655
2026-07-02T14:40:41.972863700Z 2026-07-02 14:40:41.972 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.864794513Z 2026-07-02 14:40:42.864 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.975789809Z 2026-07-02 14:40:42.975 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1698390,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-07-02T14:40:28.467845525Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-07-02T14:40:28.881193194Z *** [WARN] *** Your open file limit is currently 1024.
2026-07-02T14:40:28.881221498Z It should be set to 65000 to avoid operational disruption.
2026-07-02T14:40:28.881224568Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-07-02T14:40:28.881550800Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-07-02T14:40:28.881600193Z
2026-07-02T14:40:28.890985463Z Java 17 detected. Enabled workaround for SOLR-16463
2026-07-02T14:40:28.898465185Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-07-02T14:40:28.951650107Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-07-02T14:40:28.988393781Z WARNING: A command line option has enabled the Security Manager
2026-07-02T14:40:28.988416004Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-07-02T14:40:30.528346910Z 2026-07-02 14:40:30.525 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-07-02T14:40:30.996450459Z 2026-07-02 14:40:30.995 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-07-02T14:40:31.041659604Z 2026-07-02 14:40:31.041 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-07-02T14:40:31.052040007Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-07-02T14:40:31.052068071Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-07-02T14:40:31.052232491Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-07-02T14:40:31.052351865Z 2026-07-02 14:40:31.051 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-07-02T14:40:31.051794634Z
2026-07-02T14:40:31.055774586Z 2026-07-02 14:40:31.055 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-07-02T14:40:31.106478211Z 2026-07-02 14:40:31.105 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-07-02T14:40:31.157117541Z 2026-07-02 14:40:31.156 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.431216133Z 2026-07-02 14:40:31.430 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-07-02T14:40:31.455186365Z 2026-07-02 14:40:31.454 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:31.455298821Z 2026-07-02 14:40:31.454 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:31.455409860Z 2026-07-02 14:40:31.454 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-07-02T14:40:31.567909016Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-07-02T14:40:31.568165651Z 2026-07-02 14:40:31.567 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-07-02T14:40:31.568258191Z 2026-07-02 14:40:31.568 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-07-02T14:40:31.623844801Z 2026-07-02 14:40:31.623 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-07-02T14:40:32.697848437Z 2026-07-02 14:40:32.697 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-07-02T14:40:32.915295940Z 2026-07-02 14:40:32.914 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-07-02T14:40:32.917503700Z 2026-07-02 14:40:32.914 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-07-02T14:40:32.930843640Z 2026-07-02 14:40:32.930 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:32.937833619Z 2026-07-02 14:40:32.937 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-07-02T14:40:32.942358012Z 2026-07-02 14:40:32.941 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-07-02T14:40:32.942735450Z 2026-07-02 14:40:32.942 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-07-02T14:40:32.986591929Z 2026-07-02 14:40:32.986 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-07-02T14:40:32.996738891Z 2026-07-02 14:40:32.996 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-07-02T14:40:33.035623932Z 2026-07-02 14:40:33.034 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-07-02T14:40:33.051132226Z 2026-07-02 14:40:33.050 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-07-02T14:40:33.063028215Z 2026-07-02 14:40:33.062 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-07-02T14:40:33.074869555Z 2026-07-02 14:40:33.074 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-07-02T14:40:33.144870702Z 2026-07-02 14:40:33.144 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-07-02T14:40:33.512665504Z 2026-07-02 14:40:33.512 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-07-02T14:40:33.786030358Z 2026-07-02 14:40:33.785 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-07-02T14:40:34.079862135Z 2026-07-02 14:40:34.079 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:34.236638173Z 2026-07-02 14:40:34.235 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-07-02T14:40:34.261397120Z 2026-07-02 14:40:34.260 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_07_02.request.log
2026-07-02T14:40:34.265963788Z 2026-07-02 14:40:34.265 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-07-02T14:40:34.267476534Z 2026-07-02 14:40:34.267 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5370ms
2026-07-02T14:40:37.251924595Z 2026-07-02 14:40:37.243 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=99
2026-07-02T14:40:37.344062894Z 2026-07-02 14:40:37.343 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=20
2026-07-02T14:40:39.677471980Z 2026-07-02 14:40:39.677 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=12
2026-07-02T14:40:40.214061264Z 2026-07-02 14:40:40.212 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-07-02T14:40:40.281009350Z 2026-07-02 14:40:40.280 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-07-02T14:40:40.374284331Z 2026-07-02 14:40:40.373 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-07-02T14:40:40.877444444Z 2026-07-02 14:40:40.876 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-07-02T14:40:40.877472738Z 2026-07-02 14:40:40.877 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-07-02T14:40:40.925401098Z 2026-07-02 14:40:40.924 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-07-02T14:40:41.536689951Z 2026-07-02 14:40:41.535 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-07-02T14:40:41.566829341Z 2026-07-02 14:40:41.566 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-07-02T14:40:41.581008826Z 2026-07-02 14:40:41.580 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-07-02T14:40:41.582448423Z 2026-07-02 14:40:41.581 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-07-02T14:40:41.582574912Z 2026-07-02 14:40:41.582 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-07-02T14:40:41.641692752Z 2026-07-02 14:40:41.641 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-07-02T14:40:41.645153260Z 2026-07-02 14:40:41.644 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-07-02T14:40:41.648061587Z 2026-07-02 14:40:41.647 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-07-02T14:40:41.650192718Z 2026-07-02 14:40:41.649 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-07-02T14:40:41.666473857Z 2026-07-02 14:40:41.666 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-07-02T14:40:41.674371488Z 2026-07-02 14:40:41.673 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-07-02T14:40:41.681440738Z 2026-07-02 14:40:41.681 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-07-02T14:40:41.681684643Z 2026-07-02 14:40:41.681 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-07-02T14:40:41.689183993Z 2026-07-02 14:40:41.688 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-07-02T14:40:41.702531091Z 2026-07-02 14:40:41.701 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-07-02T14:40:41.704317786Z 2026-07-02 14:40:41.703 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-07-02T14:40:41.721402860Z 2026-07-02 14:40:41.720 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-07-02T14:40:41.721711512Z 2026-07-02 14:40:41.721 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-07-02T14:40:41.721755382Z 2026-07-02 14:40:41.721 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-07-02T14:40:41.723499918Z 2026-07-02 14:40:41.723 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-07-02T14:40:41.724549549Z 2026-07-02 14:40:41.724 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-07-02T14:40:41.725942042Z 2026-07-02 14:40:41.725 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72059597335363587-core_node5-n_0000000000
2026-07-02T14:40:41.740774482Z 2026-07-02 14:40:41.740 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-07-02T14:40:41.850245383Z 2026-07-02 14:40:41.849 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:41.855136849Z 2026-07-02 14:40:41.854 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-07-02T14:40:41.866435220Z 2026-07-02 14:40:41.866 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1654
2026-07-02T14:40:41.973425225Z 2026-07-02 14:40:41.972 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.864829959Z 2026-07-02 14:40:42.864 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.891961804Z 2026-07-02 14:40:42.891 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-07-02T14:40:42.975872729Z 2026-07-02 14:40:42.975 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-07-02T14:40:42.978816889Z 2026-07-02 14:40:42.978 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3284
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1698520,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-07-02T14:40:28.67011128Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 81 Docker SolrCloud rolling upgrade # in 28729 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Image is up to date for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 65f0f8844d165941c9ddfc17dd49077bc1d625280a6e84478a8808dc70036917
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# ac6dc439c8913c4207abda402fcf40b0552514c8b66f9032b6952414f369b8ae
# Starting solr-node2 connected to first node's ZooKeeper
# 24b8a184b892a12fbcac81a86d1bf3be9113b0511cf7235f67fb0df817e68b6b
# Started solr-node3 connected to first node's ZooKeeper
# ee1ab1fdb8c719d1472210dfc2b7e96983adf269da059f60c063dfc98f496fdd
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-07-02 14:40:39.360; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:40:39.427; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 82 allowPaths - backup # in 16808 ms
ok 83 snapshot lifecycle # in 5123 ms
ok 84 snapshot list # in 3416 ms
ok 85 snapshot list via --solr-connection # in 3375 ms
ok 86 snapshot describe # in 3345 ms
ok 87 snapshot describe via --solr-connection # in 3370 ms
ok 88 start solr with ssl # in 16374 ms
ok 89 use different hostname when not checking peer-name # in 25064 ms
ok 90 start solr with ssl and auth # in 14992 ms
ok 91 start solr with client truststore and security manager # in 16817 ms
ok 92 start solr with mTLS needed # in 46371 ms
ok 93 start solr with mTLS wanted # in 34701 ms
not ok 94 test keystore reload # in 33351 ms
# (in test file test/test_ssl.bats, line 576)
# `wait_for 30 1 solr api --solr-url "https://localhost:${SOLR_PORT}/solr/admin/info/system"' failed with status 127
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Waiting up to 180 seconds to see Solr running on port 44869 [|] [/] [-] [\] [|]
# Started Solr server on port 44869 (pid=1713085). Happy searching!
#
# INFO - 2026-07-02 14:44:09.082; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-07-02 14:44:09.085; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-07-02 14:44:09.098; org.apache.solr.common.util.EnvUtils; Deprecated system property host has been replaced by solr.host.advertise. Support for the old property will be removed in a future version of Solr.
# INFO - 2026-07-02 14:44:09.374; org.eclipse.jetty.util.ssl.SslContextFactory; x509=X509@294a6b8e(cert1,h=[localhost],a=[/127.0.0.1],w=[]) for Client@4b1d6571[provider=null,keyStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server1.keystore.p12,trustStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server1.keystore.p12]
# Waiting up to 180 seconds to see Solr running on port 44870 [|] [/] [-] [\] [|]
# Started Solr server on port 44870 (pid=1713396). Happy searching!
#
# INFO - 2026-07-02 14:44:15.157; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-07-02 14:44:15.160; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-07-02 14:44:15.172; org.apache.solr.common.util.EnvUtils; Deprecated system property host has been replaced by solr.host.advertise. Support for the old property will be removed in a future version of Solr.
# INFO - 2026-07-02 14:44:15.452; org.eclipse.jetty.util.ssl.SslContextFactory; x509=X509@294a6b8e(cert1,h=[localhost],a=[/127.0.0.1],w=[]) for Client@4b1d6571[provider=null,keyStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server2.keystore.p12,trustStore=file:///tmp/bats-run-VauQp6/test/94/ssl/server2.keystore.p12]
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_ssl.bats: line 576: wait_for: command not found
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/94-7
# Last output:
# INFO - 2026-07-02 14:44:31.876; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-07-02 14:44:31.879; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-07-02 14:44:31.892; org.apache.solr.common.util.EnvUtils; Deprecated system property host has been replaced by solr.host.advertise. Support for the old property will be removed in a future version of Solr.
# Neither --solr-connection, --zk-host or --solr-url parameters, nor SOLR_CONNECTION, ZK_HOST env var provided, so assuming solr url is https://localhost:44869.
# WARN - 2026-07-02 14:44:32.843; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:44:32.843; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 95 SOLR-11740 check 'solr stop' connection # in 15991 ms
not ok 96 stop command for single port # in 57579 ms
# (in test file test/test_start_solr.bats, line 48)
# `solr assert --started http://localhost:${SOLR2_PORT} --timeout 5000' failed with status 100
# Waiting up to 180 seconds to see Solr running on port 44869 [|] [/] [-] [\] [|]
# Started Solr server on port 44869 (pid=1715746). Happy searching!
#
# Waiting up to 180 seconds to see Solr running on port 44870 [|] [/] [-] [\] [|]
# Started Solr server on port 44870 (pid=1715880). Happy searching!
#
#
# ERROR: Solr is not running on url http://localhost:44870 after 5 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/96-2
ok 97 check stop command doesn't hang # in 221 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8471 ms
ok 99 deprecated system properties converted to modern properties # in 43818 ms
ok 100 start with custom jetty options # in 8212 ms
ok 101 -c flag prints no-op warning and still starts in cloud mode # in 9574 ms
ok 102 bootstrapping a configset # in 9520 ms
ok 103 status detects locally running solr # in 3014 ms
ok 104 status with --solr-url from user # in 1381 ms
ok 105 status with --port from user # in 1365 ms
ok 106 multiple connection options are prevented # in 816 ms
ok 107 status with invalid --solr-url from user # in 1142 ms
ok 108 status with --short format # in 767 ms
ok 109 searching solr via locally executed streaming expression # in 1786 ms
ok 110 searching solr via remotely executed streaming expression by solr-url # in 1572 ms
ok 111 searching solr via remotely executed streaming expression by solr connection # in 1255 ms
ok 112 variable interpolation # in 1684 ms
ok 113 searching solr without credentials fails with error # in 1154 ms
ok 114 --version returns Solr version # in 747 ms
ok 115 version as direct tool call still runs # in 766 ms
ok 116 short help # in 893 ms
ok 117 short help is inferred # in 753 ms
ok 118 long help # in 794 ms
ok 119 running subcommands with zk is prevented # in 773 ms
ok 120 listing out files # in 1168 ms
ok 121 connecting to solr via various solr urls and zk hosts # in 8405 ms
ok 122 copying files around # in 8185 ms
not ok 123 upconfig # in 1478 ms
# (in test file test/test_zk.bats, line 123)
# `wait_for 10 1 bash -c "curl -s 'http://localhost:${SOLR_PORT}/api/configsets' | grep -q techproducts2"' failed with status 127
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_zk.bats: line 123: wait_for: command not found
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/123-8
# Last output:
# WARN - 2026-07-02 14:48:09.702; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-07-02 14:48:09.706; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Uploading /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server/solr/configsets/sample_techproducts_configs/conf for config techproducts2 to ZooKeeper at localhost:45869
ok 124 SOLR-12429 test upconfig fails with symlink # in 1193 ms
ok 125 downconfig # in 1208 ms
ok 126 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1461 ms
ok 127 env var ZK_HOST is honored # in 2143 ms
ok 128 Cleanup # in 1917 ms
> Task :solr:packaging:integrationTests FAILED
[Incubating] Problems report is available at: file:///home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 354
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests' (registered in build file 'solr/packaging/build.gradle').
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/9.6.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 22m 40s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/h67zf5wfrehxg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 250 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/250/
No tests ran.
Build Log:
[...truncated 969 lines...]
BUILD FAILED in 29m 30s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vg4q4fekx2ryy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 249 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/249/
No tests ran.
Build Log:
[...truncated 360 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-30 12:25:19.784; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:19.789; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:20.278; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:20.278; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1715 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38671 is already being used by another process (pid: 3840819)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-30 12:25:21.482; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:25:21.485; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 18410 ms
ok 11 package with basic auth # in 12774 ms
ok 12 admin UI static content is excepted from basic auth # in 11157 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 167 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 81 ms
ok 16 compressed response can be decompressed and parsed # in 82 ms
ok 17 setting property # in 11443 ms
ok 18 short form of setting property # in 6588 ms
ok 19 connecting to solr via various connection types # in 11003 ms
ok 20 create for non cloud mode # in 8074 ms
ok 21 create for cloud mode # in 10348 ms
ok 22 multiple connection options are prevented # in 909 ms
ok 23 create collection # in 7952 ms
ok 24 create collection using solrUrl # in 5369 ms
ok 25 create collection using legacy solrUrl # in 5479 ms
ok 26 create collection using Zookeeper # in 5559 ms
ok 27 reject d option with invalid config dir # in 2810 ms
ok 28 accept d option with builtin config # in 5443 ms
ok 29 accept d option with explicit path to config # in 5394 ms
ok 30 accept n option as config name # in 5374 ms
ok 31 allow config reuse when n option specifies same config # in 9414 ms
ok 32 create multisharded collections when s provided # in 5368 ms
ok 33 create replicated collections when rf provided # in 6246 ms
ok 34 can delete collections # in 11434 ms
ok 35 can delete collections with solr-url # in 8493 ms
ok 36 can delete collections with solr-connection # in 8364 ms
ok 37 collection delete also deletes zk config # in 7642 ms
ok 38 deletes accompanying zk config with nondefault name # in 7526 ms
ok 39 delete-config option can opt to leave config in zk # in 7359 ms
ok 40 start -e cloud works with --jvm-opts # in 21994 ms
ok 41 start -e cloud works with --no-prompt # in 43838 ms
ok 42 Check export command # in 19564 ms
ok 43 export fails on non cloud mode # in 9740 ms
Waiting for Tika Server to be ready on port 38676
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16639 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16593 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16614 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 10744 ms
ok 48 healthcheck on remote cloud solr # in 10616 ms
ok 49 healthcheck errors on standalone solr # in 11622 ms
ok 50 solr help flag prints help # in 755 ms
ok 51 solr with no flags prints help # in 776 ms
ok 52 start help flag prints help # in 160 ms
ok 53 start h flag prints help # in 170 ms
ok 54 stop help flag prints help # in 183 ms
ok 55 restart help flag prints help # in 149 ms
ok 56 status help flag prints help # in 902 ms
ok 57 healthcheck help flag prints help # in 779 ms
ok 58 create help flag prints help # in 789 ms
ok 59 delete help flag prints help # in 846 ms
ok 60 zk help flag prints help # in 832 ms
ok 61 auth help flag prints help # in 807 ms
ok 62 assert help flag prints help # in 810 ms
ok 63 post help flag prints help # in 798 ms
ok 64 SQL Module # in 16163 ms
ok 65 icu collation in analysis-extras module # in 13091 ms
ok 66 Check lifecycle of sentiment classification # in 23696 ms
ok 67 lifecycle of package # in 8572 ms
ok 68 deploying and undeploying a collection level package # in 13671 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 16327 ms
ok 71 Random placement plugin using ENV # in 16180 ms
ok 72 Check help command # in 2440 ms
ok 73 basic post with a type specified # in 5521 ms
ok 74 basic post with collection instead of url parameter # in 4076 ms
ok 75 basic post with solr-url and collection # in 4013 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 4052 ms
ok 77 basic post WITHOUT a type specified # in 11873 ms
ok 78 crawling a directory as a dry-run # in 790 ms
ok 79 crawling a directory # in 4499 ms
ok 80 skipcommit and optimize and delete # in 4070 ms
ok 81 args mode # in 7133 ms
ok 82 stdin mode # in 3525 ms
ok 83 verbose echo the Solr response # in 3909 ms
ok 84 post solr log into solr via cli # in 9887 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 27 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-30T12:35:54.550909185Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-30T12:35:54.550951947Z It should be set to 65000 to avoid operational disruption.
2026-06-30T12:35:54.550955008Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-30T12:35:54.551258270Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-30T12:35:54.551288184Z
2026-06-30T12:35:54.564941461Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-30T12:35:54.575824741Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-30T12:35:54.640864625Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-30T12:35:54.690288316Z WARNING: A command line option has enabled the Security Manager
2026-06-30T12:35:54.690316937Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-30T12:35:57.256469140Z 2026-06-30 12:35:57.250 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-30T12:35:57.804325541Z 2026-06-30 12:35:57.803 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-30T12:35:57.855314415Z 2026-06-30 12:35:57.854 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-30T12:35:57.917530071Z 2026-06-30 12:35:57.916 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-30T12:35:57.917897726Z 2026-06-30 12:35:57.917 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-30T12:35:57.917947402Z 2026-06-30 12:35:57.917 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-30T12:35:57.917955776Z 2026-06-30 12:35:57.917 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-30T12:35:57.917291391Z
2026-06-30T12:35:57.921813618Z 2026-06-30 12:35:57.921 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-30T12:35:58.017549573Z 2026-06-30 12:35:58.016 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-30T12:35:58.117251670Z 2026-06-30 12:35:58.116 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-30T12:35:58.117273064Z 2026-06-30 12:35:58.116 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-30T12:35:58.225009411Z 2026-06-30 12:35:58.224 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-30T12:35:59.836348789Z 2026-06-30 12:35:59.835 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-30T12:36:00.343050701Z 2026-06-30 12:36:00.342 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-30T12:36:00.356538268Z 2026-06-30 12:36:00.356 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-30T12:36:00.356586392Z 2026-06-30 12:36:00.356 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-30T12:36:00.362842261Z 2026-06-30 12:36:00.362 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-30T12:36:00.362860204Z 2026-06-30 12:36:00.362 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-30T12:36:00.451185734Z 2026-06-30 12:36:00.450 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-30T12:36:00.865523017Z 2026-06-30 12:36:00.864 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-30T12:36:00.895291722Z 2026-06-30 12:36:00.894 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:00.952192320Z 2026-06-30 12:36:00.951 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-30T12:36:00.981591030Z 2026-06-30 12:36:00.980 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:00.981614125Z 2026-06-30 12:36:00.981 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:01.116294798Z 2026-06-30 12:36:01.115 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-30T12:36:01.144317758Z 2026-06-30 12:36:01.143 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:01.156305071Z 2026-06-30 12:36:01.155 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-30T12:36:01.163157173Z 2026-06-30 12:36:01.162 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72211905013022720-solr-node1:8983_solr-n_0000000000) starting
2026-06-30T12:36:01.256719268Z 2026-06-30 12:36:01.256 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-30T12:36:01.258241836Z 2026-06-30 12:36:01.257 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-30T12:36:01.269882857Z 2026-06-30 12:36:01.269 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-30T12:36:01.339563200Z 2026-06-30 12:36:01.338 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-30T12:36:02.098931877Z 2026-06-30 12:36:02.098 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-30T12:36:02.448914594Z 2026-06-30 12:36:02.448 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-30T12:36:02.987660217Z 2026-06-30 12:36:02.987 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:03.474744324Z 2026-06-30 12:36:03.473 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-30T12:36:03.537341806Z 2026-06-30 12:36:03.536 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_30.request.log
2026-06-30T12:36:03.544208562Z 2026-06-30 12:36:03.543 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-30T12:36:03.546358308Z 2026-06-30 12:36:03.545 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @8971ms
2026-06-30T12:36:04.059633144Z 2026-06-30 12:36:04.059 INFO (qtp985324122-28-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=185
2026-06-30T12:36:04.391558628Z 2026-06-30 12:36:04.391 INFO (qtp985324122-97-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19
2026-06-30T12:36:09.266627422Z 2026-06-30 12:36:09.266 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-30T12:36:09.873312912Z 2026-06-30 12:36:09.872 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-30T12:36:16.156824845Z 2026-06-30 12:36:16.155 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-06-30T12:36:16.250284588Z 2026-06-30 12:36:16.249 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=33
2026-06-30T12:36:16.287984776Z 2026-06-30 12:36:16.287 INFO (qtp985324122-28-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=3
2026-06-30T12:36:16.769002614Z 2026-06-30 12:36:16.768 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-30T12:36:16.808289161Z 2026-06-30 12:36:16.807 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-30T12:36:16.826741457Z 2026-06-30 12:36:16.826 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:16.941832654Z 2026-06-30 12:36:16.941 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-30T12:36:16.941859648Z "core":"test-collection_shard3_replica_n1",
2026-06-30T12:36:16.941863040Z "node_name":"solr-node1:8983_solr",
2026-06-30T12:36:16.941865492Z "base_url":"http://solr-node1:8983/solr",
2026-06-30T12:36:16.941867792Z "collection":"test-collection",
2026-06-30T12:36:16.941870315Z "shard":"shard3",
2026-06-30T12:36:16.941872391Z "state":"down",
2026-06-30T12:36:16.941874441Z "type":"NRT",
2026-06-30T12:36:16.941876437Z "operation":"ADDREPLICA",
2026-06-30T12:36:16.941879022Z "waitForFinalState":"false"}
2026-06-30T12:36:16.955543252Z 2026-06-30 12:36:16.954 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-30T12:36:16.955571028Z "core":"test-collection_shard2_replica_n2",
2026-06-30T12:36:16.955574050Z "node_name":"solr-node3:8985_solr",
2026-06-30T12:36:16.955577393Z "base_url":"http://solr-node3:8985/solr",
2026-06-30T12:36:16.955579678Z "collection":"test-collection",
2026-06-30T12:36:16.955581761Z "shard":"shard2",
2026-06-30T12:36:16.955583755Z "state":"down",
2026-06-30T12:36:16.955585944Z "type":"NRT",
2026-06-30T12:36:16.955587823Z "operation":"ADDREPLICA",
2026-06-30T12:36:16.955589757Z "waitForFinalState":"false"}
2026-06-30T12:36:16.964320064Z 2026-06-30 12:36:16.963 INFO (OverseerStateUpdate-72211905013022720-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-30T12:36:16.964344747Z "core":"test-collection_shard1_replica_n4",
2026-06-30T12:36:16.964347602Z "node_name":"solr-node2:8984_solr",
2026-06-30T12:36:16.964364080Z "base_url":"http://solr-node2:8984/solr",
2026-06-30T12:36:16.964366345Z "collection":"test-collection",
2026-06-30T12:36:16.964368350Z "shard":"shard1",
2026-06-30T12:36:16.964370283Z "state":"down",
2026-06-30T12:36:16.964372276Z "type":"NRT",
2026-06-30T12:36:16.964374313Z "operation":"ADDREPLICA",
2026-06-30T12:36:16.964376251Z "waitForFinalState":"false"}
2026-06-30T12:36:17.078848605Z 2026-06-30 12:36:17.078 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:17.224735665Z 2026-06-30 12:36:17.224 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-30T12:36:17.385039436Z 2026-06-30 12:36:17.384 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-30T12:36:17.524382511Z 2026-06-30 12:36:17.522 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-30T12:36:18.239618842Z 2026-06-30 12:36:18.238 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-30T12:36:18.239648283Z 2026-06-30 12:36:18.239 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-30T12:36:18.301296188Z 2026-06-30 12:36:18.300 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-06-30T12:36:19.768834907Z 2026-06-30 12:36:19.767 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:19.836794016Z 2026-06-30 12:36:19.836 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-30T12:36:19.841868799Z 2026-06-30 12:36:19.840 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-30T12:36:19.842778426Z 2026-06-30 12:36:19.841 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-30T12:36:19.842946327Z 2026-06-30 12:36:19.842 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-30T12:36:19.915598214Z 2026-06-30 12:36:19.915 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-30T12:36:19.922732992Z 2026-06-30 12:36:19.922 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-30T12:36:19.924832029Z 2026-06-30 12:36:19.924 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-30T12:36:19.926637572Z 2026-06-30 12:36:19.926 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-30T12:36:19.950377724Z 2026-06-30 12:36:19.949 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-30T12:36:19.975689865Z 2026-06-30 12:36:19.975 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-30T12:36:19.993646315Z 2026-06-30 12:36:19.993 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-30T12:36:19.993673597Z 2026-06-30 12:36:19.993 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-30T12:36:20.002596419Z 2026-06-30 12:36:20.002 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-30T12:36:20.012026938Z 2026-06-30 12:36:20.011 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-30T12:36:20.016318963Z 2026-06-30 12:36:20.015 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-30T12:36:20.033276793Z 2026-06-30 12:36:20.032 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-30T12:36:20.033299248Z 2026-06-30 12:36:20.033 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-30T12:36:20.033355758Z 2026-06-30 12:36:20.033 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-06-30T12:36:20.034866722Z 2026-06-30 12:36:20.034 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-30T12:36:20.035350675Z 2026-06-30 12:36:20.035 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-06-30T12:36:20.036542316Z 2026-06-30 12:36:20.036 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72211905013022720-core_node3-n_0000000000
2026-06-30T12:36:20.041638624Z 2026-06-30 12:36:20.041 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-06-30T12:36:20.148181385Z 2026-06-30 12:36:20.147 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.152110319Z 2026-06-30 12:36:20.151 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-30T12:36:20.159275633Z 2026-06-30 12:36:20.158 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2936
2026-06-30T12:36:20.260022506Z 2026-06-30 12:36:20.259 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3874439,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-30T12:35:52.85964948Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-30T12:36:05.352428393Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-30T12:36:05.352474801Z It should be set to 65000 to avoid operational disruption.
2026-06-30T12:36:05.352480161Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-30T12:36:05.352731650Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-30T12:36:05.352739751Z
2026-06-30T12:36:05.362455677Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-30T12:36:05.372009841Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-30T12:36:05.424723388Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-30T12:36:05.462127653Z WARNING: A command line option has enabled the Security Manager
2026-06-30T12:36:05.462173345Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-30T12:36:07.663109673Z 2026-06-30 12:36:07.659 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-30T12:36:07.965196332Z 2026-06-30 12:36:07.964 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-30T12:36:07.995956641Z 2026-06-30 12:36:07.995 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-30T12:36:08.005144306Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-30T12:36:08.005518241Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-30T12:36:08.005526899Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-30T12:36:08.005530641Z 2026-06-30 12:36:08.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-30T12:36:08.004959307Z
2026-06-30T12:36:08.009050609Z 2026-06-30 12:36:08.008 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-30T12:36:08.034199901Z 2026-06-30 12:36:08.033 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-30T12:36:08.071472978Z 2026-06-30 12:36:08.070 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:08.357547855Z 2026-06-30 12:36:08.357 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-30T12:36:08.382666565Z 2026-06-30 12:36:08.382 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:08.382700753Z 2026-06-30 12:36:08.382 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:08.382822804Z 2026-06-30 12:36:08.382 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:08.493201133Z 2026-06-30 12:36:08.492 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-30T12:36:08.493350081Z 2026-06-30 12:36:08.493 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-30T12:36:08.493501086Z 2026-06-30 12:36:08.493 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-30T12:36:08.530009957Z 2026-06-30 12:36:08.529 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-30T12:36:09.586780668Z 2026-06-30 12:36:09.586 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-30T12:36:09.743722871Z 2026-06-30 12:36:09.743 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-30T12:36:09.745825393Z 2026-06-30 12:36:09.743 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-30T12:36:09.754710710Z 2026-06-30 12:36:09.754 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.760131196Z 2026-06-30 12:36:09.759 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-30T12:36:09.764446794Z 2026-06-30 12:36:09.764 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:09.764641550Z 2026-06-30 12:36:09.764 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:09.800962873Z 2026-06-30 12:36:09.800 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-30T12:36:09.809582523Z 2026-06-30 12:36:09.809 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-30T12:36:09.836268950Z 2026-06-30 12:36:09.835 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.848651945Z 2026-06-30 12:36:09.848 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-30T12:36:09.864199171Z 2026-06-30 12:36:09.863 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-30T12:36:09.874498816Z 2026-06-30 12:36:09.874 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-30T12:36:09.928701897Z 2026-06-30 12:36:09.928 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-30T12:36:10.210828671Z 2026-06-30 12:36:10.210 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-30T12:36:10.399466477Z 2026-06-30 12:36:10.398 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-30T12:36:10.667609046Z 2026-06-30 12:36:10.667 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:10.794966884Z 2026-06-30 12:36:10.794 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-30T12:36:10.800969044Z 2026-06-30 12:36:10.800 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_30.request.log
2026-06-30T12:36:10.804461011Z 2026-06-30 12:36:10.803 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-30T12:36:10.805258608Z 2026-06-30 12:36:10.805 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5434ms
2026-06-30T12:36:11.030462368Z 2026-06-30 12:36:11.029 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=58
2026-06-30T12:36:11.160001881Z 2026-06-30 12:36:11.159 INFO (qtp985324122-56-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-06-30T12:36:17.224299106Z 2026-06-30 12:36:17.223 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-30T12:36:17.291832663Z 2026-06-30 12:36:17.289 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-30T12:36:17.401378993Z 2026-06-30 12:36:17.400 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-30T12:36:17.917359349Z 2026-06-30 12:36:17.916 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-30T12:36:17.917394256Z 2026-06-30 12:36:17.916 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-30T12:36:17.949569753Z 2026-06-30 12:36:17.949 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-30T12:36:18.729512404Z 2026-06-30 12:36:18.728 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:18.755532434Z 2026-06-30 12:36:18.755 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-30T12:36:18.770900992Z 2026-06-30 12:36:18.770 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-30T12:36:18.772301223Z 2026-06-30 12:36:18.771 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-30T12:36:18.772375185Z 2026-06-30 12:36:18.772 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-30T12:36:18.836414113Z 2026-06-30 12:36:18.835 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-30T12:36:18.840540057Z 2026-06-30 12:36:18.840 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-30T12:36:18.843240280Z 2026-06-30 12:36:18.842 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-30T12:36:18.845993119Z 2026-06-30 12:36:18.845 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-30T12:36:18.867526846Z 2026-06-30 12:36:18.866 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-30T12:36:18.877353482Z 2026-06-30 12:36:18.876 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-30T12:36:18.885125923Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-30T12:36:18.885148125Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-30T12:36:18.906970171Z 2026-06-30 12:36:18.906 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-30T12:36:18.930691200Z 2026-06-30 12:36:18.929 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-30T12:36:18.932400239Z 2026-06-30 12:36:18.932 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-30T12:36:18.950225987Z 2026-06-30 12:36:18.949 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-30T12:36:18.950286988Z 2026-06-30 12:36:18.949 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-30T12:36:18.950351077Z 2026-06-30 12:36:18.950 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-30T12:36:18.952272822Z 2026-06-30 12:36:18.951 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-30T12:36:18.953258334Z 2026-06-30 12:36:18.953 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-30T12:36:18.954926918Z 2026-06-30 12:36:18.954 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72211905013022724-core_node6-n_0000000000
2026-06-30T12:36:18.973141714Z 2026-06-30 12:36:18.972 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-30T12:36:19.083681932Z 2026-06-30 12:36:19.083 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:19.089548646Z 2026-06-30 12:36:19.089 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-30T12:36:19.103633828Z 2026-06-30 12:36:19.103 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1881
2026-06-30T12:36:19.206266158Z 2026-06-30 12:36:19.205 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.148495885Z 2026-06-30 12:36:20.147 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.259948258Z 2026-06-30 12:36:20.259 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3874843,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-30T12:36:04.796102086Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-30T12:36:05.352697068Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-30T12:36:05.352722043Z It should be set to 65000 to avoid operational disruption.
2026-06-30T12:36:05.352725204Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-30T12:36:05.353083003Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-30T12:36:05.353095323Z
2026-06-30T12:36:05.362672573Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-30T12:36:05.371923394Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-30T12:36:05.424665619Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-30T12:36:05.462181551Z WARNING: A command line option has enabled the Security Manager
2026-06-30T12:36:05.462202669Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-30T12:36:07.018495886Z 2026-06-30 12:36:07.014 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-30T12:36:07.338992303Z 2026-06-30 12:36:07.338 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-30T12:36:07.371255765Z 2026-06-30 12:36:07.370 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-30T12:36:07.381430818Z 2026-06-30 12:36:07.380 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-30T12:36:07.381501108Z 2026-06-30 12:36:07.381 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-30T12:36:07.381621174Z 2026-06-30 12:36:07.381 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-30T12:36:07.381726936Z 2026-06-30 12:36:07.381 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-30T12:36:07.381199368Z
2026-06-30T12:36:07.384783909Z 2026-06-30 12:36:07.384 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-30T12:36:07.421295016Z 2026-06-30 12:36:07.420 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-30T12:36:07.489356333Z 2026-06-30 12:36:07.488 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:07.663368557Z 2026-06-30 12:36:07.662 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-30T12:36:07.684918498Z 2026-06-30 12:36:07.684 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:07.685024769Z 2026-06-30 12:36:07.684 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:07.685162403Z 2026-06-30 12:36:07.684 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-30T12:36:07.800958684Z 2026-06-30 12:36:07.800 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-30T12:36:07.802034867Z 2026-06-30 12:36:07.800 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-30T12:36:07.802235307Z 2026-06-30 12:36:07.801 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-30T12:36:07.856849195Z 2026-06-30 12:36:07.856 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-30T12:36:08.899447678Z 2026-06-30 12:36:08.898 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-30T12:36:09.123366398Z 2026-06-30 12:36:09.122 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-30T12:36:09.125436362Z 2026-06-30 12:36:09.122 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-30T12:36:09.138662035Z 2026-06-30 12:36:09.138 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.146377911Z 2026-06-30 12:36:09.145 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-30T12:36:09.149916037Z 2026-06-30 12:36:09.149 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-30T12:36:09.150133790Z 2026-06-30 12:36:09.149 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-30T12:36:09.189254720Z 2026-06-30 12:36:09.188 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-30T12:36:09.197620347Z 2026-06-30 12:36:09.197 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-30T12:36:09.231503799Z 2026-06-30 12:36:09.230 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-30T12:36:09.243707563Z 2026-06-30 12:36:09.243 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-30T12:36:09.255457775Z 2026-06-30 12:36:09.254 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-30T12:36:09.267734741Z 2026-06-30 12:36:09.267 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-30T12:36:09.335603339Z 2026-06-30 12:36:09.334 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-30T12:36:09.655132527Z 2026-06-30 12:36:09.654 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-30T12:36:09.873642195Z 2026-06-30 12:36:09.873 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-30T12:36:09.891950947Z 2026-06-30 12:36:09.891 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-30T12:36:10.217638422Z 2026-06-30 12:36:10.217 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:10.368035789Z 2026-06-30 12:36:10.367 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-30T12:36:10.375329305Z 2026-06-30 12:36:10.374 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_30.request.log
2026-06-30T12:36:10.381266526Z 2026-06-30 12:36:10.380 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-30T12:36:10.382174004Z 2026-06-30 12:36:10.381 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5011ms
2026-06-30T12:36:13.860142370Z 2026-06-30 12:36:13.855 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=116
2026-06-30T12:36:13.962891801Z 2026-06-30 12:36:13.962 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=22
2026-06-30T12:36:16.718824764Z 2026-06-30 12:36:16.718 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=10
2026-06-30T12:36:17.224004464Z 2026-06-30 12:36:17.222 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-30T12:36:17.301565699Z 2026-06-30 12:36:17.300 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-30T12:36:17.408899788Z 2026-06-30 12:36:17.408 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-30T12:36:17.919108397Z 2026-06-30 12:36:17.918 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-30T12:36:17.919157776Z 2026-06-30 12:36:17.918 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-30T12:36:17.949576739Z 2026-06-30 12:36:17.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-06-30T12:36:18.729459412Z 2026-06-30 12:36:18.728 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-30T12:36:18.755590987Z 2026-06-30 12:36:18.755 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-30T12:36:18.770904133Z 2026-06-30 12:36:18.770 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-30T12:36:18.772104995Z 2026-06-30 12:36:18.771 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-30T12:36:18.772172428Z 2026-06-30 12:36:18.771 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-30T12:36:18.836413094Z 2026-06-30 12:36:18.835 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-30T12:36:18.840415249Z 2026-06-30 12:36:18.839 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-30T12:36:18.844296865Z 2026-06-30 12:36:18.843 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-30T12:36:18.855184622Z 2026-06-30 12:36:18.854 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-30T12:36:18.870195349Z 2026-06-30 12:36:18.869 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-30T12:36:18.879038416Z 2026-06-30 12:36:18.878 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-30T12:36:18.885164852Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-30T12:36:18.885244874Z 2026-06-30 12:36:18.884 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-30T12:36:18.907288751Z 2026-06-30 12:36:18.906 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-30T12:36:18.930588718Z 2026-06-30 12:36:18.929 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-30T12:36:18.932238116Z 2026-06-30 12:36:18.931 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-30T12:36:18.949747160Z 2026-06-30 12:36:18.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-30T12:36:18.949776009Z 2026-06-30 12:36:18.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-30T12:36:18.949793068Z 2026-06-30 12:36:18.949 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-06-30T12:36:18.951812284Z 2026-06-30 12:36:18.951 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-30T12:36:18.952369594Z 2026-06-30 12:36:18.952 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-06-30T12:36:18.954284507Z 2026-06-30 12:36:18.954 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72211905013022723-core_node5-n_0000000000
2026-06-30T12:36:18.975646137Z 2026-06-30 12:36:18.975 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-06-30T12:36:19.083723174Z 2026-06-30 12:36:19.083 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:19.089161179Z 2026-06-30 12:36:19.088 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-30T12:36:19.102369616Z 2026-06-30 12:36:19.101 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1880
2026-06-30T12:36:19.206448907Z 2026-06-30 12:36:19.205 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.148296989Z 2026-06-30 12:36:20.147 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.173438126Z 2026-06-30 12:36:20.172 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-30T12:36:20.260054617Z 2026-06-30 12:36:20.259 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-30T12:36:20.262618990Z 2026-06-30 12:36:20.262 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3522
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3874970,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-30T12:36:05.001479307Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 31169 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Image is up to date for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 649b7c42f74ea036f61c7cf952c066c83501dd51fb60557609b30eea9737c0a3
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# a0f85b1c1772d934d438563f26b5e9faa9baa2ae57025c13b15db1a97fd1539f
# Starting solr-node2 connected to first node's ZooKeeper
# 069309e2a2e2dab82a50fc0d975525773d88c7995f809dd0a189ce0c46de2e09
# Started solr-node3 connected to first node's ZooKeeper
# bdaa1dfa287b895579204a77d08082948b265a849c3b5f2cc4d6ad98b5524a64
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-30 12:36:16.355; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-30 12:36:16.438; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 16371 ms
ok 87 snapshot lifecycle # in 5043 ms
ok 88 snapshot list # in 3434 ms
ok 89 snapshot list via --solr-connection # in 3467 ms
ok 90 snapshot describe # in 3506 ms
ok 91 snapshot describe via --solr-connection # in 3619 ms
ok 92 start solr with ssl # in 17316 ms
ok 93 use different hostname when not checking peer-name # in 25267 ms
ok 94 start solr with ssl and auth # in 15169 ms
ok 95 start solr with client truststore and security manager # in 17086 ms
ok 96 start solr with mTLS needed # in 47267 ms
ok 97 start solr with mTLS wanted # in 35975 ms
ok 98 test keystore reload # in 60702 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 16316 ms
ok 100 stop command for single port # in 18749 ms
ok 101 check stop command doesn't hang # in 225 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8310 ms
ok 103 deprecated system properties converted to modern properties # in 195424 ms
ok 104 start with custom jetty options # in 8138 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 9133 ms
ok 106 bootstrapping a configset # in 9544 ms
ok 107 status detects locally running solr # in 9361 ms
ok 108 status with --solr-url from user # in 8075 ms
ok 109 status with --port from user # in 8034 ms
ok 110 multiple connection options are prevented # in 850 ms
ok 111 status with invalid --solr-url from user # in 1237 ms
ok 112 status with --short format # in 195837 ms
ok 113 searching solr via locally executed streaming expression # in 1741 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1507 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1197 ms
ok 116 variable interpolation # in 1751 ms
ok 117 searching solr without credentials fails with error # in 1164 ms
ok 118 --version returns Solr version # in 731 ms
ok 119 version as direct tool call still runs # in 729 ms
ok 120 short help # in 752 ms
ok 121 short help is inferred # in 763 ms
ok 122 long help # in 768 ms
ok 123 running subcommands with zk is prevented # in 740 ms
ok 124 listing out files # in 2129 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 9191 ms
ok 126 copying files around # in 10834 ms
ok 127 upconfig # in 2566 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1154 ms
ok 129 downconfig # in 1191 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1451 ms
ok 131 env var ZK_HOST is honored # in 3173 ms
ok 132 Cleanup # in 1844 ms
> Task :solr:packaging:integrationTests FAILED
[Incubating] Problems report is available at: file:///home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests' (registered in build file 'solr/packaging/build.gradle').
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/9.6.0/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 35s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/wtd3utwi3v6hk
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 248 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/248/
No tests ran.
Build Log:
[...truncated 1004 lines...]
BUILD FAILED in 30m 27s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/emaopxwt3jqki
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 247 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/247/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-28 13:24:53.710; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-28 13:24:53.714; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-28 13:24:54.205; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-28 13:24:54.206; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1448 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46267 is already being used by another process (pid: 2712210)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-28 13:24:55.515; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-28 13:24:55.519; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 19521 ms
ok 11 package with basic auth # in 13212 ms
ok 12 admin UI static content is excepted from basic auth # in 11529 ms
ok 13 nothing # in 11 ms
ok 14 server does not compress response without Accept-Encoding header # in 168 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 100 ms
ok 16 compressed response can be decompressed and parsed # in 86 ms
ok 17 setting property # in 13516 ms
ok 18 short form of setting property # in 6994 ms
ok 19 connecting to solr via various connection types # in 11756 ms
ok 20 create for non cloud mode # in 8602 ms
ok 21 create for cloud mode # in 10674 ms
ok 22 multiple connection options are prevented # in 964 ms
ok 23 create collection # in 8990 ms
ok 24 create collection using solrUrl # in 5654 ms
ok 25 create collection using legacy solrUrl # in 5611 ms
ok 26 create collection using Zookeeper # in 5875 ms
ok 27 reject d option with invalid config dir # in 2895 ms
ok 28 accept d option with builtin config # in 5619 ms
ok 29 accept d option with explicit path to config # in 5550 ms
ok 30 accept n option as config name # in 5577 ms
ok 31 allow config reuse when n option specifies same config # in 9770 ms
ok 32 create multisharded collections when s provided # in 5747 ms
ok 33 create replicated collections when rf provided # in 6522 ms
ok 34 can delete collections # in 12124 ms
ok 35 can delete collections with solr-url # in 8993 ms
ok 36 can delete collections with solr-connection # in 8597 ms
ok 37 collection delete also deletes zk config # in 7981 ms
ok 38 deletes accompanying zk config with nondefault name # in 8006 ms
ok 39 delete-config option can opt to leave config in zk # in 7679 ms
ok 40 start -e cloud works with --jvm-opts # in 23296 ms
ok 41 start -e cloud works with --no-prompt # in 44383 ms
ok 42 Check export command # in 20641 ms
ok 43 export fails on non cloud mode # in 9981 ms
Waiting for Tika Server to be ready on port 46272
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16776 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16639 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16682 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 11076 ms
ok 48 healthcheck on remote cloud solr # in 11115 ms
ok 49 healthcheck errors on standalone solr # in 12452 ms
ok 50 solr help flag prints help # in 829 ms
ok 51 solr with no flags prints help # in 819 ms
ok 52 start help flag prints help # in 167 ms
ok 53 start h flag prints help # in 163 ms
ok 54 stop help flag prints help # in 158 ms
ok 55 restart help flag prints help # in 161 ms
ok 56 status help flag prints help # in 878 ms
ok 57 healthcheck help flag prints help # in 829 ms
ok 58 create help flag prints help # in 827 ms
ok 59 delete help flag prints help # in 871 ms
ok 60 zk help flag prints help # in 852 ms
ok 61 auth help flag prints help # in 832 ms
ok 62 assert help flag prints help # in 836 ms
ok 63 post help flag prints help # in 834 ms
ok 64 SQL Module # in 17449 ms
ok 65 icu collation in analysis-extras module # in 13779 ms
ok 66 Check lifecycle of sentiment classification # in 24529 ms
ok 67 lifecycle of package # in 9160 ms
ok 68 deploying and undeploying a collection level package # in 14443 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 17266 ms
ok 71 Random placement plugin using ENV # in 17247 ms
ok 72 Check help command # in 2676 ms
ok 73 basic post with a type specified # in 6013 ms
ok 74 basic post with collection instead of url parameter # in 4496 ms
ok 75 basic post with solr-url and collection # in 4409 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 4330 ms
ok 77 basic post WITHOUT a type specified # in 13048 ms
ok 78 crawling a directory as a dry-run # in 853 ms
ok 79 crawling a directory # in 4827 ms
ok 80 skipcommit and optimize and delete # in 4214 ms
ok 81 args mode # in 7612 ms
ok 82 stdin mode # in 3711 ms
ok 83 verbose echo the Solr response # in 4256 ms
ok 84 post solr log into solr via cli # in 10745 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-28T13:36:07.539438184Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-28T13:36:07.539478755Z It should be set to 65000 to avoid operational disruption.
2026-06-28T13:36:07.539481917Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-28T13:36:07.539781475Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-28T13:36:07.539792845Z
2026-06-28T13:36:07.551073984Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-28T13:36:07.559888889Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-28T13:36:07.605340559Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-28T13:36:07.639510463Z WARNING: A command line option has enabled the Security Manager
2026-06-28T13:36:07.639540653Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-28T13:36:08.904684420Z 2026-06-28 13:36:08.901 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-28T13:36:09.194715929Z 2026-06-28 13:36:09.194 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-28T13:36:09.221361743Z 2026-06-28 13:36:09.220 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-28T13:36:09.229580238Z 2026-06-28 13:36:09.228 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-28T13:36:09.229784382Z 2026-06-28 13:36:09.228 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-28T13:36:09.230030325Z 2026-06-28 13:36:09.228 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-28T13:36:09.230237650Z 2026-06-28 13:36:09.228 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-28T13:36:09.228914300Z
2026-06-28T13:36:09.232757949Z 2026-06-28 13:36:09.232 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-28T13:36:09.255915768Z 2026-06-28 13:36:09.255 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-28T13:36:09.270574147Z 2026-06-28 13:36:09.269 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-28T13:36:09.270596286Z 2026-06-28 13:36:09.269 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-28T13:36:09.308430012Z 2026-06-28 13:36:09.307 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-28T13:36:10.084324211Z 2026-06-28 13:36:10.083 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-28T13:36:10.344278280Z 2026-06-28 13:36:10.343 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-28T13:36:10.350028653Z 2026-06-28 13:36:10.349 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-28T13:36:10.350192270Z 2026-06-28 13:36:10.349 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-28T13:36:10.353034217Z 2026-06-28 13:36:10.352 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-28T13:36:10.353056998Z 2026-06-28 13:36:10.352 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-28T13:36:10.390732252Z 2026-06-28 13:36:10.390 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-28T13:36:10.859940802Z 2026-06-28 13:36:10.858 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-28T13:36:10.874824248Z 2026-06-28 13:36:10.874 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-28T13:36:10.925971862Z 2026-06-28 13:36:10.925 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-28T13:36:11.020984500Z 2026-06-28 13:36:11.019 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-28T13:36:11.021246302Z 2026-06-28 13:36:11.019 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-28T13:36:11.176089981Z 2026-06-28 13:36:11.175 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-28T13:36:11.206686098Z 2026-06-28 13:36:11.206 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-28T13:36:11.223157566Z 2026-06-28 13:36:11.222 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-28T13:36:11.230273536Z 2026-06-28 13:36:11.229 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72200816973905920-solr-node1:8983_solr-n_0000000000) starting
2026-06-28T13:36:11.308505164Z 2026-06-28 13:36:11.307 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-28T13:36:11.309469463Z 2026-06-28 13:36:11.309 INFO (OverseerStateUpdate-72200816973905920-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-28T13:36:11.321450110Z 2026-06-28 13:36:11.320 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-28T13:36:11.370159651Z 2026-06-28 13:36:11.369 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-28T13:36:11.827056014Z 2026-06-28 13:36:11.826 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-28T13:36:12.013541339Z 2026-06-28 13:36:12.013 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-28T13:36:12.295731583Z 2026-06-28 13:36:12.295 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-28T13:36:12.435948701Z 2026-06-28 13:36:12.435 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-28T13:36:12.441957700Z 2026-06-28 13:36:12.441 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_28.request.log
2026-06-28T13:36:12.446443937Z 2026-06-28 13:36:12.445 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-28T13:36:12.487366548Z 2026-06-28 13:36:12.486 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4927ms
2026-06-28T13:36:12.669777164Z 2026-06-28 13:36:12.669 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=51
2026-06-28T13:36:12.732821299Z 2026-06-28 13:36:12.732 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-06-28T13:36:16.124848460Z 2026-06-28 13:36:16.124 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-28T13:36:16.440961283Z 2026-06-28 13:36:16.440 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-28T13:36:20.438515384Z 2026-06-28 13:36:20.438 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-06-28T13:36:20.466363717Z 2026-06-28 13:36:20.465 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-06-28T13:36:20.495054995Z 2026-06-28 13:36:20.494 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-06-28T13:36:20.736817691Z 2026-06-28 13:36:20.735 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=6
2026-06-28T13:36:20.765417908Z 2026-06-28 13:36:20.765 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-28T13:36:20.784593977Z 2026-06-28 13:36:20.784 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-28T13:36:20.790491099Z 2026-06-28 13:36:20.790 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:20.839319425Z 2026-06-28 13:36:20.838 INFO (OverseerStateUpdate-72200816973905920-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-28T13:36:20.839339994Z "core":"test-collection_shard3_replica_n1",
2026-06-28T13:36:20.839343213Z "node_name":"solr-node1:8983_solr",
2026-06-28T13:36:20.839345381Z "base_url":"http://solr-node1:8983/solr",
2026-06-28T13:36:20.839347495Z "collection":"test-collection",
2026-06-28T13:36:20.839349641Z "shard":"shard3",
2026-06-28T13:36:20.839351730Z "state":"down",
2026-06-28T13:36:20.839353747Z "type":"NRT",
2026-06-28T13:36:20.839355830Z "operation":"ADDREPLICA",
2026-06-28T13:36:20.839357789Z "waitForFinalState":"false"}
2026-06-28T13:36:20.847394919Z 2026-06-28 13:36:20.846 INFO (OverseerStateUpdate-72200816973905920-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-28T13:36:20.847420288Z "core":"test-collection_shard2_replica_n2",
2026-06-28T13:36:20.847423202Z "node_name":"solr-node3:8985_solr",
2026-06-28T13:36:20.847425414Z "base_url":"http://solr-node3:8985/solr",
2026-06-28T13:36:20.847427674Z "collection":"test-collection",
2026-06-28T13:36:20.847429731Z "shard":"shard2",
2026-06-28T13:36:20.847431841Z "state":"down",
2026-06-28T13:36:20.847433834Z "type":"NRT",
2026-06-28T13:36:20.847435811Z "operation":"ADDREPLICA",
2026-06-28T13:36:20.847437883Z "waitForFinalState":"false"}
2026-06-28T13:36:20.858456720Z 2026-06-28 13:36:20.857 INFO (OverseerStateUpdate-72200816973905920-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-28T13:36:20.858489456Z "core":"test-collection_shard1_replica_n4",
2026-06-28T13:36:20.858492412Z "node_name":"solr-node2:8984_solr",
2026-06-28T13:36:20.858494544Z "base_url":"http://solr-node2:8984/solr",
2026-06-28T13:36:20.858496525Z "collection":"test-collection",
2026-06-28T13:36:20.858498481Z "shard":"shard1",
2026-06-28T13:36:20.858500393Z "state":"down",
2026-06-28T13:36:20.858502278Z "type":"NRT",
2026-06-28T13:36:20.858504324Z "operation":"ADDREPLICA",
2026-06-28T13:36:20.858506292Z "waitForFinalState":"false"}
2026-06-28T13:36:20.972694362Z 2026-06-28 13:36:20.972 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:21.062312630Z 2026-06-28 13:36:21.061 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-28T13:36:21.165112712Z 2026-06-28 13:36:21.164 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-28T13:36:21.213747945Z 2026-06-28 13:36:21.213 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-28T13:36:21.544680830Z 2026-06-28 13:36:21.544 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-28T13:36:21.544703728Z 2026-06-28 13:36:21.544 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-28T13:36:21.579726809Z 2026-06-28 13:36:21.579 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-06-28T13:36:22.369822464Z 2026-06-28 13:36:22.368 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-28T13:36:22.407898332Z 2026-06-28 13:36:22.406 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-28T13:36:22.415950172Z 2026-06-28 13:36:22.415 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-28T13:36:22.417363755Z 2026-06-28 13:36:22.416 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-28T13:36:22.417461812Z 2026-06-28 13:36:22.416 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-28T13:36:22.491952769Z 2026-06-28 13:36:22.491 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-28T13:36:22.496737667Z 2026-06-28 13:36:22.496 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-28T13:36:22.503251987Z 2026-06-28 13:36:22.502 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-28T13:36:22.505995348Z 2026-06-28 13:36:22.504 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-28T13:36:22.526179802Z 2026-06-28 13:36:22.525 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-28T13:36:22.535416601Z 2026-06-28 13:36:22.535 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-28T13:36:22.544875166Z 2026-06-28 13:36:22.544 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-28T13:36:22.545016524Z 2026-06-28 13:36:22.544 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-28T13:36:22.554905614Z 2026-06-28 13:36:22.554 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-28T13:36:22.575857237Z 2026-06-28 13:36:22.575 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-28T13:36:22.577704034Z 2026-06-28 13:36:22.577 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-28T13:36:22.604258309Z 2026-06-28 13:36:22.603 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-28T13:36:22.604419987Z 2026-06-28 13:36:22.603 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-28T13:36:22.604433060Z 2026-06-28 13:36:22.604 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-06-28T13:36:22.605968736Z 2026-06-28 13:36:22.605 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-28T13:36:22.606737753Z 2026-06-28 13:36:22.606 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-06-28T13:36:22.608096595Z 2026-06-28 13:36:22.607 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72200816973905920-core_node3-n_0000000000
2026-06-28T13:36:22.614889002Z 2026-06-28 13:36:22.614 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-06-28T13:36:22.719895730Z 2026-06-28 13:36:22.719 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:22.726757929Z 2026-06-28 13:36:22.726 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-28T13:36:22.742397679Z 2026-06-28 13:36:22.741 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1682
2026-06-28T13:36:22.753125660Z 2026-06-28 13:36:22.752 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-28T13:36:22.842691929Z 2026-06-28 13:36:22.842 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:22.846769678Z 2026-06-28 13:36:22.846 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2097
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2746146,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-28T13:36:07.30852805Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-28T13:36:13.049647037Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-28T13:36:13.049682417Z It should be set to 65000 to avoid operational disruption.
2026-06-28T13:36:13.049685758Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-28T13:36:13.050013365Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-28T13:36:13.050026301Z
2026-06-28T13:36:13.060746850Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-28T13:36:13.069336663Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-28T13:36:13.113759727Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-28T13:36:13.153561217Z WARNING: A command line option has enabled the Security Manager
2026-06-28T13:36:13.153585885Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-28T13:36:14.319307615Z 2026-06-28 13:36:14.315 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-28T13:36:14.623581639Z 2026-06-28 13:36:14.622 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-28T13:36:14.651783101Z 2026-06-28 13:36:14.650 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-28T13:36:14.659046834Z 2026-06-28 13:36:14.658 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-28T13:36:14.659956032Z 2026-06-28 13:36:14.658 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-28T13:36:14.659975380Z 2026-06-28 13:36:14.658 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-28T13:36:14.659978628Z 2026-06-28 13:36:14.658 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-28T13:36:14.658616029Z
2026-06-28T13:36:14.662089049Z 2026-06-28 13:36:14.661 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-28T13:36:14.685760391Z 2026-06-28 13:36:14.685 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-28T13:36:14.712061463Z 2026-06-28 13:36:14.711 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-28T13:36:14.876684153Z 2026-06-28 13:36:14.876 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-28T13:36:14.895970270Z 2026-06-28 13:36:14.895 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-28T13:36:14.896209175Z 2026-06-28 13:36:14.895 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-28T13:36:14.896359566Z 2026-06-28 13:36:14.895 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-28T13:36:15.007452635Z 2026-06-28 13:36:15.006 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-28T13:36:15.007701414Z 2026-06-28 13:36:15.007 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-28T13:36:15.007790378Z 2026-06-28 13:36:15.007 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-28T13:36:15.045543164Z 2026-06-28 13:36:15.044 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-28T13:36:15.816073900Z 2026-06-28 13:36:15.815 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-28T13:36:15.995345100Z 2026-06-28 13:36:15.992 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-28T13:36:15.997548830Z 2026-06-28 13:36:15.993 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-28T13:36:16.004613988Z 2026-06-28 13:36:16.004 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-28T13:36:16.011567941Z 2026-06-28 13:36:16.010 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-28T13:36:16.016634524Z 2026-06-28 13:36:16.016 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-28T13:36:16.016792065Z 2026-06-28 13:36:16.016 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-28T13:36:16.052530109Z 2026-06-28 13:36:16.051 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-28T13:36:16.061997630Z 2026-06-28 13:36:16.061 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-28T13:36:16.084012619Z 2026-06-28 13:36:16.083 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-28T13:36:16.100307838Z 2026-06-28 13:36:16.099 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-28T13:36:16.113173602Z 2026-06-28 13:36:16.112 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-28T13:36:16.126291146Z 2026-06-28 13:36:16.125 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-28T13:36:16.240136669Z 2026-06-28 13:36:16.238 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-28T13:36:16.441670781Z 2026-06-28 13:36:16.440 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-28T13:36:16.575626680Z 2026-06-28 13:36:16.574 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-28T13:36:16.789717228Z 2026-06-28 13:36:16.788 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-28T13:36:17.094061285Z 2026-06-28 13:36:17.092 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-28T13:36:17.245095966Z 2026-06-28 13:36:17.244 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-28T13:36:17.252371543Z 2026-06-28 13:36:17.251 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_28.request.log
2026-06-28T13:36:17.258240199Z 2026-06-28 13:36:17.256 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-28T13:36:17.258964150Z 2026-06-28 13:36:17.258 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4190ms
2026-06-28T13:36:17.432951807Z 2026-06-28 13:36:17.432 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=49
2026-06-28T13:36:17.488244788Z 2026-06-28 13:36:17.487 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-06-28T13:36:21.062496471Z 2026-06-28 13:36:21.061 INFO (qtp985324122-59-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-28T13:36:21.122251551Z 2026-06-28 13:36:21.121 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-28T13:36:21.183600427Z 2026-06-28 13:36:21.182 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-28T13:36:21.515407702Z 2026-06-28 13:36:21.514 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-28T13:36:21.515432195Z 2026-06-28 13:36:21.515 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-28T13:36:21.535942007Z 2026-06-28 13:36:21.535 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-28T13:36:22.039933891Z 2026-06-28 13:36:22.039 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-28T13:36:22.060443104Z 2026-06-28 13:36:22.059 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-28T13:36:22.063950078Z 2026-06-28 13:36:22.063 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-28T13:36:22.065184561Z 2026-06-28 13:36:22.064 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-28T13:36:22.065200780Z 2026-06-28 13:36:22.064 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-28T13:36:22.172578717Z 2026-06-28 13:36:22.172 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-28T13:36:22.176581683Z 2026-06-28 13:36:22.176 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-28T13:36:22.180140002Z 2026-06-28 13:36:22.179 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-28T13:36:22.182398599Z 2026-06-28 13:36:22.182 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-28T13:36:22.195537180Z 2026-06-28 13:36:22.194 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-28T13:36:22.203650720Z 2026-06-28 13:36:22.203 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-28T13:36:22.210553023Z 2026-06-28 13:36:22.210 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-28T13:36:22.210573893Z 2026-06-28 13:36:22.210 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-28T13:36:22.219492823Z 2026-06-28 13:36:22.218 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-28T13:36:22.235836129Z 2026-06-28 13:36:22.234 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-28T13:36:22.237539933Z 2026-06-28 13:36:22.237 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-28T13:36:22.256758044Z 2026-06-28 13:36:22.256 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-28T13:36:22.256780546Z 2026-06-28 13:36:22.256 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-28T13:36:22.256865833Z 2026-06-28 13:36:22.256 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-28T13:36:22.258582813Z 2026-06-28 13:36:22.258 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-28T13:36:22.259588512Z 2026-06-28 13:36:22.259 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-28T13:36:22.260881137Z 2026-06-28 13:36:22.260 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72200816973905923-core_node6-n_0000000000
2026-06-28T13:36:22.273667169Z 2026-06-28 13:36:22.273 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-28T13:36:22.383088557Z 2026-06-28 13:36:22.382 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:22.387394557Z 2026-06-28 13:36:22.386 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-28T13:36:22.399437428Z 2026-06-28 13:36:22.399 INFO (qtp985324122-59-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1338
2026-06-28T13:36:22.501168498Z 2026-06-28 13:36:22.499 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 209] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:22.720208791Z 2026-06-28 13:36:22.719 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:22.843276129Z 2026-06-28 13:36:22.842 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2746543,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-28T13:36:12.831103404Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-28T13:36:13.278493121Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-28T13:36:13.278536525Z It should be set to 65000 to avoid operational disruption.
2026-06-28T13:36:13.278541510Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-28T13:36:13.278852190Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-28T13:36:13.278859308Z
2026-06-28T13:36:13.289146488Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-28T13:36:13.297896227Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-28T13:36:13.344454947Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-28T13:36:13.377732640Z WARNING: A command line option has enabled the Security Manager
2026-06-28T13:36:13.377757390Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-28T13:36:14.582170591Z 2026-06-28 13:36:14.577 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-28T13:36:14.885658502Z 2026-06-28 13:36:14.884 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-28T13:36:14.913490600Z 2026-06-28 13:36:14.912 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-28T13:36:14.921248552Z 2026-06-28 13:36:14.920 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-28T13:36:14.921361806Z 2026-06-28 13:36:14.920 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-28T13:36:14.921441414Z 2026-06-28 13:36:14.920 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-28T13:36:14.921603661Z 2026-06-28 13:36:14.920 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-28T13:36:14.920851485Z
2026-06-28T13:36:14.924487710Z 2026-06-28 13:36:14.924 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-28T13:36:14.948021256Z 2026-06-28 13:36:14.947 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-28T13:36:14.974295493Z 2026-06-28 13:36:14.973 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-28T13:36:15.135616144Z 2026-06-28 13:36:15.134 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-28T13:36:15.154750189Z 2026-06-28 13:36:15.154 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-28T13:36:15.154949770Z 2026-06-28 13:36:15.154 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-28T13:36:15.155084276Z 2026-06-28 13:36:15.154 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-28T13:36:15.265718384Z 2026-06-28 13:36:15.265 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-28T13:36:15.265884701Z 2026-06-28 13:36:15.265 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-28T13:36:15.266056265Z 2026-06-28 13:36:15.265 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-28T13:36:15.301455292Z 2026-06-28 13:36:15.300 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-28T13:36:16.155873685Z 2026-06-28 13:36:16.155 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-28T13:36:16.308108190Z 2026-06-28 13:36:16.307 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-28T13:36:16.310368337Z 2026-06-28 13:36:16.307 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-28T13:36:16.319399323Z 2026-06-28 13:36:16.318 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-28T13:36:16.325582295Z 2026-06-28 13:36:16.324 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-28T13:36:16.330516687Z 2026-06-28 13:36:16.330 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-28T13:36:16.330624949Z 2026-06-28 13:36:16.330 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-28T13:36:16.371905515Z 2026-06-28 13:36:16.371 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-28T13:36:16.381400028Z 2026-06-28 13:36:16.380 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-28T13:36:16.406361136Z 2026-06-28 13:36:16.405 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-28T13:36:16.419765851Z 2026-06-28 13:36:16.419 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-28T13:36:16.431515535Z 2026-06-28 13:36:16.431 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-28T13:36:16.442230824Z 2026-06-28 13:36:16.441 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-28T13:36:16.487793712Z 2026-06-28 13:36:16.487 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-28T13:36:16.785274279Z 2026-06-28 13:36:16.784 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-28T13:36:16.970625708Z 2026-06-28 13:36:16.970 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-28T13:36:17.267510993Z 2026-06-28 13:36:17.266 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-28T13:36:17.402637813Z 2026-06-28 13:36:17.401 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-28T13:36:17.408544174Z 2026-06-28 13:36:17.408 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_28.request.log
2026-06-28T13:36:17.412405376Z 2026-06-28 13:36:17.411 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-28T13:36:17.413312566Z 2026-06-28 13:36:17.412 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4116ms
2026-06-28T13:36:19.180159214Z 2026-06-28 13:36:19.173 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=139
2026-06-28T13:36:19.270031145Z 2026-06-28 13:36:19.269 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=27
2026-06-28T13:36:21.066457837Z 2026-06-28 13:36:21.065 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-28T13:36:21.159950322Z 2026-06-28 13:36:21.159 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-28T13:36:21.272048024Z 2026-06-28 13:36:21.271 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-28T13:36:21.693741037Z 2026-06-28 13:36:21.693 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-28T13:36:21.693765997Z 2026-06-28 13:36:21.693 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-28T13:36:21.717317421Z 2026-06-28 13:36:21.716 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-06-28T13:36:22.361817214Z 2026-06-28 13:36:22.361 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-28T13:36:22.386366859Z 2026-06-28 13:36:22.385 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-28T13:36:22.394204747Z 2026-06-28 13:36:22.393 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-28T13:36:22.398251116Z 2026-06-28 13:36:22.394 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-28T13:36:22.398270042Z 2026-06-28 13:36:22.397 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-28T13:36:22.458654858Z 2026-06-28 13:36:22.458 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-28T13:36:22.463082169Z 2026-06-28 13:36:22.462 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-28T13:36:22.465735560Z 2026-06-28 13:36:22.465 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-28T13:36:22.468891962Z 2026-06-28 13:36:22.468 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-28T13:36:22.482661797Z 2026-06-28 13:36:22.482 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-28T13:36:22.491026568Z 2026-06-28 13:36:22.490 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-28T13:36:22.498112621Z 2026-06-28 13:36:22.497 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-28T13:36:22.498520008Z 2026-06-28 13:36:22.497 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-28T13:36:22.508710382Z 2026-06-28 13:36:22.508 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-28T13:36:22.523020933Z 2026-06-28 13:36:22.522 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-28T13:36:22.525546906Z 2026-06-28 13:36:22.525 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-28T13:36:22.543050830Z 2026-06-28 13:36:22.542 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-28T13:36:22.543302330Z 2026-06-28 13:36:22.542 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-28T13:36:22.543320478Z 2026-06-28 13:36:22.542 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-06-28T13:36:22.544642679Z 2026-06-28 13:36:22.544 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-28T13:36:22.545644030Z 2026-06-28 13:36:22.545 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-06-28T13:36:22.546983199Z 2026-06-28 13:36:22.546 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72200816973905924-core_node5-n_0000000000
2026-06-28T13:36:22.569596301Z 2026-06-28 13:36:22.569 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-06-28T13:36:22.720547891Z 2026-06-28 13:36:22.720 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-28T13:36:22.725691509Z 2026-06-28 13:36:22.725 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-28T13:36:22.741816093Z 2026-06-28 13:36:22.741 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1678
2026-06-28T13:36:22.843282907Z 2026-06-28 13:36:22.842 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2746729,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-28T13:36:13.040668007Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 30277 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 69a09c96e8bf: Waiting
# 3aa9120e46f5: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 66b76b382631: Waiting
# 4f4fb700ef54: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Pulling fs layer
# 9d5a0b6cb36c: Pulling fs layer
# 49518c28c0b3: Pulling fs layer
# 41162acdcc29: Pulling fs layer
# 2ed0fd558ca3: Pulling fs layer
# 2488415fda1c: Pulling fs layer
# 09511c8fb203: Pulling fs layer
# a0c57fb89c4f: Pulling fs layer
# d0641dcc41d7: Pulling fs layer
# 315722fd9570: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 41162acdcc29: Waiting
# 2ed0fd558ca3: Waiting
# 2488415fda1c: Waiting
# 09511c8fb203: Waiting
# a0c57fb89c4f: Waiting
# 315722fd9570: Waiting
# 4f4fb700ef54: Waiting
# d0641dcc41d7: Waiting
# 40d16f30db40: Verifying Checksum
# 40d16f30db40: Download complete
# 9d5a0b6cb36c: Verifying Checksum
# 9d5a0b6cb36c: Download complete
# 41162acdcc29: Verifying Checksum
# 41162acdcc29: Download complete
# 2ed0fd558ca3: Download complete
# 09511c8fb203: Verifying Checksum
# 09511c8fb203: Download complete
# a0c57fb89c4f: Download complete
# 49518c28c0b3: Verifying Checksum
# 49518c28c0b3: Download complete
# d0641dcc41d7: Verifying Checksum
# d0641dcc41d7: Download complete
# 315722fd9570: Verifying Checksum
# 315722fd9570: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 2488415fda1c: Verifying Checksum
# 2488415fda1c: Download complete
# 40d16f30db40: Pull complete
# 9d5a0b6cb36c: Pull complete
# 49518c28c0b3: Pull complete
# 41162acdcc29: Pull complete
# 2ed0fd558ca3: Pull complete
# 2488415fda1c: Pull complete
# 09511c8fb203: Pull complete
# a0c57fb89c4f: Pull complete
# d0641dcc41d7: Pull complete
# 315722fd9570: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 4d3c843a4df37b20def9b41ba3626be43b69ada825462be05f6bac1c335f8771
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 99c55f7ac6834bff851837fe6f8a80e3bc32986561adcd7bf1cc6f047e8d16df
# Starting solr-node2 connected to first node's ZooKeeper
# eb8f91e796d14efedcbf4b1489a667a046a5fc810414d5a02eb0bc016f37d3a9
# Started solr-node3 connected to first node's ZooKeeper
# 80c1902d7f0892d94773df9cf518f00ba4baf1fccd10881571f7f3da3fd78dc2
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-28 13:36:20.525; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-28 13:36:20.566; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 16748 ms
ok 87 snapshot lifecycle # in 5270 ms
ok 88 snapshot list # in 3675 ms
ok 89 snapshot list via --solr-connection # in 3730 ms
ok 90 snapshot describe # in 3705 ms
ok 91 snapshot describe via --solr-connection # in 3663 ms
ok 92 start solr with ssl # in 17674 ms
ok 93 use different hostname when not checking peer-name # in 26935 ms
ok 94 start solr with ssl and auth # in 16039 ms
ok 95 start solr with client truststore and security manager # in 18137 ms
ok 96 start solr with mTLS needed # in 50342 ms
ok 97 start solr with mTLS wanted # in 38325 ms
ok 98 test keystore reload # in 63134 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 16849 ms
ok 100 stop command for single port # in 19359 ms
ok 101 check stop command doesn't hang # in 244 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8698 ms
not ok 103 deprecated system properties converted to modern properties # in 195672 ms
# (from function `assert_file_contains' in file .gradle/node/node_modules/bats-file/src/file.bash, line 566,
# in test file test/test_start_solr.bats, line 86)
# `assert_file_contains "${SOLR_LOGS_DIR}/solr.log" 'You are passing in deprecated system property disable.config.edit and should upgrade to using solr.api.config.edit.enabled instead.'' failed
# Waiting up to 180 seconds to see Solr running on port 46267 [|] [/] [-] [\] [|]
# Started Solr server on port 46267 (pid=2764542). Happy searching!
#
#
# -- file does not contain regex --
# path : /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/solr.log
# --
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-5
ok 104 start with custom jetty options # in 8896 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 10065 ms
not ok 106 bootstrapping a configset # in 10075 ms
# (from function `config_exists' in file test/bats_helper.bash, line 84,
# in test file test/test_start_solr.bats, line 116)
# `config_exists "techproducts"' failed
# Waiting up to 180 seconds to see Solr running on port 46267 [|] [/] [-] [\] [|]
# Started Solr server on port 46267 (pid=2767279). Happy searching!
#
# WARN - 2026-06-28 13:45:29.289; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-28 13:45:29.293; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-8
ok 107 status detects locally running solr # in 10357 ms
ok 108 status with --solr-url from user # in 8692 ms
ok 109 status with --port from user # in 8711 ms
ok 110 multiple connection options are prevented # in 1102 ms
ok 111 status with invalid --solr-url from user # in 1426 ms
ok 112 status with --short format # in 196655 ms
ok 113 searching solr via locally executed streaming expression # in 1898 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1728 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1349 ms
ok 116 variable interpolation # in 1865 ms
ok 117 searching solr without credentials fails with error # in 1233 ms
ok 118 --version returns Solr version # in 821 ms
ok 119 version as direct tool call still runs # in 819 ms
ok 120 short help # in 918 ms
ok 121 short help is inferred # in 867 ms
ok 122 long help # in 841 ms
ok 123 running subcommands with zk is prevented # in 801 ms
ok 124 listing out files # in 2255 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 10115 ms
ok 126 copying files around # in 11809 ms
ok 127 upconfig # in 2757 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1299 ms
ok 129 downconfig # in 1324 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1561 ms
ok 131 env var ZK_HOST is honored # in 3358 ms
ok 132 Cleanup # in 1926 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 57s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/oeqtasufdylmu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 246 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/246/
No tests ran.
Build Log:
[...truncated 1000 lines...]
BUILD FAILED in 29m 28s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/4tdkvllpljkku
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 245 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/245/
No tests ran.
Build Log:
[...truncated 1005 lines...]
BUILD FAILED in 28m 51s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/3ogxaj5gaxgem
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 244 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/244/
No tests ran.
Build Log:
[...truncated 1004 lines...]
BUILD FAILED in 28m 51s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vf5vqmhbiq5ne
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 243 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/243/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-24 12:38:52.647; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-24 12:38:52.651; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-24 12:38:53.146; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-24 12:38:53.146; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 11029 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 40685 is already being used by another process (pid: 2536087)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-24 12:38:54.390; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-24 12:38:54.394; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = ConnectionLoss for /security.json
ok 10 post, postlogs and export with basic auth # in 18735 ms
ok 11 package with basic auth # in 12859 ms
ok 12 admin UI static content is excepted from basic auth # in 11299 ms
ok 13 nothing # in 13 ms
ok 14 server does not compress response without Accept-Encoding header # in 168 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 82 ms
ok 16 compressed response can be decompressed and parsed # in 84 ms
ok 17 setting property # in 12538 ms
ok 18 short form of setting property # in 6982 ms
ok 19 connecting to solr via various connection types # in 11486 ms
ok 20 create for non cloud mode # in 8491 ms
ok 21 create for cloud mode # in 10481 ms
ok 22 multiple connection options are prevented # in 946 ms
ok 23 create collection # in 8434 ms
ok 24 create collection using solrUrl # in 5750 ms
ok 25 create collection using legacy solrUrl # in 5500 ms
ok 26 create collection using Zookeeper # in 5815 ms
ok 27 reject d option with invalid config dir # in 2847 ms
ok 28 accept d option with builtin config # in 5704 ms
ok 29 accept d option with explicit path to config # in 5568 ms
ok 30 accept n option as config name # in 5429 ms
ok 31 allow config reuse when n option specifies same config # in 9842 ms
ok 32 create multisharded collections when s provided # in 5404 ms
ok 33 create replicated collections when rf provided # in 6388 ms
ok 34 can delete collections # in 12069 ms
ok 35 can delete collections with solr-url # in 8881 ms
ok 36 can delete collections with solr-connection # in 8562 ms
ok 37 collection delete also deletes zk config # in 7936 ms
ok 38 deletes accompanying zk config with nondefault name # in 8037 ms
ok 39 delete-config option can opt to leave config in zk # in 7491 ms
ok 40 start -e cloud works with --jvm-opts # in 23009 ms
ok 41 start -e cloud works with --no-prompt # in 45672 ms
ok 42 Check export command # in 20545 ms
ok 43 export fails on non cloud mode # in 9858 ms
Waiting for Tika Server to be ready on port 40690
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16648 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16668 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16630 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 10883 ms
ok 48 healthcheck on remote cloud solr # in 10436 ms
ok 49 healthcheck errors on standalone solr # in 12183 ms
ok 50 solr help flag prints help # in 764 ms
ok 51 solr with no flags prints help # in 779 ms
ok 52 start help flag prints help # in 155 ms
ok 53 start h flag prints help # in 157 ms
ok 54 stop help flag prints help # in 150 ms
ok 55 restart help flag prints help # in 156 ms
ok 56 status help flag prints help # in 826 ms
ok 57 healthcheck help flag prints help # in 781 ms
ok 58 create help flag prints help # in 796 ms
ok 59 delete help flag prints help # in 836 ms
ok 60 zk help flag prints help # in 817 ms
ok 61 auth help flag prints help # in 785 ms
ok 62 assert help flag prints help # in 810 ms
ok 63 post help flag prints help # in 888 ms
ok 64 SQL Module # in 16947 ms
ok 65 icu collation in analysis-extras module # in 13068 ms
ok 66 Check lifecycle of sentiment classification # in 24364 ms
ok 67 lifecycle of package # in 8719 ms
ok 68 deploying and undeploying a collection level package # in 14073 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 16398 ms
ok 71 Random placement plugin using ENV # in 16227 ms
ok 72 Check help command # in 2554 ms
ok 73 basic post with a type specified # in 5908 ms
ok 74 basic post with collection instead of url parameter # in 4098 ms
ok 75 basic post with solr-url and collection # in 4220 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 4212 ms
ok 77 basic post WITHOUT a type specified # in 12100 ms
ok 78 crawling a directory as a dry-run # in 862 ms
ok 79 crawling a directory # in 4775 ms
ok 80 skipcommit and optimize and delete # in 4041 ms
ok 81 args mode # in 7349 ms
ok 82 stdin mode # in 3592 ms
ok 83 verbose echo the Solr response # in 4130 ms
ok 84 post solr log into solr via cli # in 10230 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-24T12:50:02.602157180Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-24T12:50:02.602201591Z It should be set to 65000 to avoid operational disruption.
2026-06-24T12:50:02.602205032Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-24T12:50:02.602570572Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-24T12:50:02.602582884Z
2026-06-24T12:50:02.614223433Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-24T12:50:02.622537514Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-24T12:50:02.668637442Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-24T12:50:02.700003577Z WARNING: A command line option has enabled the Security Manager
2026-06-24T12:50:02.700027459Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-24T12:50:03.994557352Z 2026-06-24 12:50:03.991 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-24T12:50:04.258527055Z 2026-06-24 12:50:04.257 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-24T12:50:04.284037622Z 2026-06-24 12:50:04.283 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-24T12:50:04.292165744Z 2026-06-24 12:50:04.291 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-24T12:50:04.292384628Z 2026-06-24 12:50:04.291 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-24T12:50:04.292579679Z 2026-06-24 12:50:04.291 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-24T12:50:04.292708472Z 2026-06-24 12:50:04.291 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-24T12:50:04.291718583Z
2026-06-24T12:50:04.295594185Z 2026-06-24 12:50:04.295 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-24T12:50:04.318459686Z 2026-06-24 12:50:04.317 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-24T12:50:04.332736225Z 2026-06-24 12:50:04.332 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-24T12:50:04.333031939Z 2026-06-24 12:50:04.332 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-24T12:50:04.369761053Z 2026-06-24 12:50:04.369 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-24T12:50:05.114067530Z 2026-06-24 12:50:05.113 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-24T12:50:05.362793281Z 2026-06-24 12:50:05.362 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-24T12:50:05.367836382Z 2026-06-24 12:50:05.367 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-24T12:50:05.367854812Z 2026-06-24 12:50:05.367 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-24T12:50:05.370408800Z 2026-06-24 12:50:05.369 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-24T12:50:05.370435554Z 2026-06-24 12:50:05.370 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-24T12:50:05.407363009Z 2026-06-24 12:50:05.406 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-24T12:50:05.877319007Z 2026-06-24 12:50:05.876 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-24T12:50:05.892176914Z 2026-06-24 12:50:05.891 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-24T12:50:05.932753883Z 2026-06-24 12:50:05.932 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-24T12:50:05.965484429Z 2026-06-24 12:50:05.964 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-24T12:50:05.965596570Z 2026-06-24 12:50:05.964 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-24T12:50:06.091513908Z 2026-06-24 12:50:06.090 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-24T12:50:06.114647834Z 2026-06-24 12:50:06.114 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-24T12:50:06.127648600Z 2026-06-24 12:50:06.127 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-24T12:50:06.137838592Z 2026-06-24 12:50:06.137 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72177986523693056-solr-node1:8983_solr-n_0000000000) starting
2026-06-24T12:50:06.202985634Z 2026-06-24 12:50:06.202 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-24T12:50:06.206367993Z 2026-06-24 12:50:06.206 INFO (OverseerStateUpdate-72177986523693056-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-24T12:50:06.215322401Z 2026-06-24 12:50:06.214 INFO (OverseerStateUpdate-72177986523693056-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-24T12:50:06.257930162Z 2026-06-24 12:50:06.257 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-24T12:50:06.645104577Z 2026-06-24 12:50:06.644 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-24T12:50:06.834663439Z 2026-06-24 12:50:06.834 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-24T12:50:07.120230700Z 2026-06-24 12:50:07.119 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-24T12:50:07.267293009Z 2026-06-24 12:50:07.266 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-24T12:50:07.274523591Z 2026-06-24 12:50:07.274 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_24.request.log
2026-06-24T12:50:07.279416533Z 2026-06-24 12:50:07.278 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-24T12:50:07.281969560Z 2026-06-24 12:50:07.281 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4660ms
2026-06-24T12:50:07.522745531Z 2026-06-24 12:50:07.522 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=48
2026-06-24T12:50:07.575298814Z 2026-06-24 12:50:07.574 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-06-24T12:50:10.965209111Z 2026-06-24 12:50:10.964 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-24T12:50:11.045021108Z 2026-06-24 12:50:11.044 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-24T12:50:15.063090849Z 2026-06-24 12:50:15.062 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-06-24T12:50:15.091090421Z 2026-06-24 12:50:15.090 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-06-24T12:50:15.126712825Z 2026-06-24 12:50:15.126 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-06-24T12:50:15.526632794Z 2026-06-24 12:50:15.525 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=5
2026-06-24T12:50:15.554942374Z 2026-06-24 12:50:15.554 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-24T12:50:15.570623955Z 2026-06-24 12:50:15.570 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-24T12:50:15.579476770Z 2026-06-24 12:50:15.578 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:15.626535932Z 2026-06-24 12:50:15.626 INFO (OverseerStateUpdate-72177986523693056-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-24T12:50:15.626558320Z "core":"test-collection_shard3_replica_n1",
2026-06-24T12:50:15.626562159Z "node_name":"solr-node1:8983_solr",
2026-06-24T12:50:15.626565141Z "base_url":"http://solr-node1:8983/solr",
2026-06-24T12:50:15.626568375Z "collection":"test-collection",
2026-06-24T12:50:15.626571163Z "shard":"shard3",
2026-06-24T12:50:15.626573978Z "state":"down",
2026-06-24T12:50:15.626576600Z "type":"NRT",
2026-06-24T12:50:15.626579167Z "operation":"ADDREPLICA",
2026-06-24T12:50:15.626581899Z "waitForFinalState":"false"}
2026-06-24T12:50:15.634206804Z 2026-06-24 12:50:15.633 INFO (OverseerStateUpdate-72177986523693056-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-24T12:50:15.634226874Z "core":"test-collection_shard2_replica_n2",
2026-06-24T12:50:15.634229274Z "node_name":"solr-node3:8985_solr",
2026-06-24T12:50:15.634231766Z "base_url":"http://solr-node3:8985/solr",
2026-06-24T12:50:15.634233900Z "collection":"test-collection",
2026-06-24T12:50:15.634236177Z "shard":"shard2",
2026-06-24T12:50:15.634238153Z "state":"down",
2026-06-24T12:50:15.634240203Z "type":"NRT",
2026-06-24T12:50:15.634242134Z "operation":"ADDREPLICA",
2026-06-24T12:50:15.634245153Z "waitForFinalState":"false"}
2026-06-24T12:50:15.642358693Z 2026-06-24 12:50:15.641 INFO (OverseerStateUpdate-72177986523693056-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-24T12:50:15.642393585Z "core":"test-collection_shard1_replica_n4",
2026-06-24T12:50:15.642396560Z "node_name":"solr-node2:8984_solr",
2026-06-24T12:50:15.642398771Z "base_url":"http://solr-node2:8984/solr",
2026-06-24T12:50:15.642400740Z "collection":"test-collection",
2026-06-24T12:50:15.642402830Z "shard":"shard1",
2026-06-24T12:50:15.642404800Z "state":"down",
2026-06-24T12:50:15.642406707Z "type":"NRT",
2026-06-24T12:50:15.642408703Z "operation":"ADDREPLICA",
2026-06-24T12:50:15.642410686Z "waitForFinalState":"false"}
2026-06-24T12:50:15.756734129Z 2026-06-24 12:50:15.756 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:15.856106102Z 2026-06-24 12:50:15.855 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-24T12:50:15.949661268Z 2026-06-24 12:50:15.948 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-24T12:50:16.026551167Z 2026-06-24 12:50:16.025 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-24T12:50:16.476891973Z 2026-06-24 12:50:16.476 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-24T12:50:16.477005046Z 2026-06-24 12:50:16.476 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-24T12:50:16.510762084Z 2026-06-24 12:50:16.510 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-06-24T12:50:17.664752441Z 2026-06-24 12:50:17.663 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-24T12:50:17.719872361Z 2026-06-24 12:50:17.718 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-24T12:50:17.731060555Z 2026-06-24 12:50:17.730 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-24T12:50:17.733720499Z 2026-06-24 12:50:17.733 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-24T12:50:17.733794028Z 2026-06-24 12:50:17.733 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-24T12:50:17.811174338Z 2026-06-24 12:50:17.810 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-24T12:50:17.813949430Z 2026-06-24 12:50:17.813 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-24T12:50:17.816148287Z 2026-06-24 12:50:17.815 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-24T12:50:17.818256026Z 2026-06-24 12:50:17.818 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-24T12:50:17.841061898Z 2026-06-24 12:50:17.840 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-24T12:50:17.850565040Z 2026-06-24 12:50:17.850 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-24T12:50:17.859209354Z 2026-06-24 12:50:17.858 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-24T12:50:17.859238884Z 2026-06-24 12:50:17.858 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-24T12:50:17.870498690Z 2026-06-24 12:50:17.870 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-24T12:50:17.878953985Z 2026-06-24 12:50:17.878 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-24T12:50:17.881280875Z 2026-06-24 12:50:17.880 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-24T12:50:17.897475725Z 2026-06-24 12:50:17.897 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-24T12:50:17.897503757Z 2026-06-24 12:50:17.897 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-24T12:50:17.897530049Z 2026-06-24 12:50:17.897 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-06-24T12:50:17.898891660Z 2026-06-24 12:50:17.898 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-24T12:50:17.899493379Z 2026-06-24 12:50:17.899 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-06-24T12:50:17.900592550Z 2026-06-24 12:50:17.900 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72177986523693056-core_node3-n_0000000000
2026-06-24T12:50:17.907786530Z 2026-06-24 12:50:17.907 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-06-24T12:50:18.013775889Z 2026-06-24 12:50:18.013 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:18.017623113Z 2026-06-24 12:50:18.017 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-24T12:50:18.026562571Z 2026-06-24 12:50:18.026 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2171
2026-06-24T12:50:18.036848804Z 2026-06-24 12:50:18.036 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-24T12:50:18.125977513Z 2026-06-24 12:50:18.125 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:18.128972243Z 2026-06-24 12:50:18.128 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2591
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2567588,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-24T12:50:02.365514837Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-24T12:50:07.884518158Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-24T12:50:07.884544549Z It should be set to 65000 to avoid operational disruption.
2026-06-24T12:50:07.884547685Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-24T12:50:07.884833505Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-24T12:50:07.884847359Z
2026-06-24T12:50:07.895751345Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-24T12:50:07.903884678Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-24T12:50:07.944112086Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-24T12:50:07.983308884Z WARNING: A command line option has enabled the Security Manager
2026-06-24T12:50:07.983328404Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-24T12:50:09.206105833Z 2026-06-24 12:50:09.202 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-24T12:50:09.489623922Z 2026-06-24 12:50:09.488 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-24T12:50:09.516394901Z 2026-06-24 12:50:09.515 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-24T12:50:09.524499084Z 2026-06-24 12:50:09.523 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-24T12:50:09.524580893Z 2026-06-24 12:50:09.524 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-24T12:50:09.524893580Z 2026-06-24 12:50:09.524 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-24T12:50:09.525156525Z 2026-06-24 12:50:09.524 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-24T12:50:09.524162200Z
2026-06-24T12:50:09.527662572Z 2026-06-24 12:50:09.527 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-24T12:50:09.555529979Z 2026-06-24 12:50:09.554 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-24T12:50:09.582758259Z 2026-06-24 12:50:09.582 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-24T12:50:09.742517185Z 2026-06-24 12:50:09.741 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-24T12:50:09.766261810Z 2026-06-24 12:50:09.765 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-24T12:50:09.766357831Z 2026-06-24 12:50:09.765 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-24T12:50:09.766462367Z 2026-06-24 12:50:09.765 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-24T12:50:09.880092497Z 2026-06-24 12:50:09.878 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-24T12:50:09.880122653Z 2026-06-24 12:50:09.879 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-24T12:50:09.880311668Z 2026-06-24 12:50:09.879 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-24T12:50:09.922255614Z 2026-06-24 12:50:09.921 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-24T12:50:10.746990585Z 2026-06-24 12:50:10.746 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-24T12:50:10.926574644Z 2026-06-24 12:50:10.925 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-24T12:50:10.928271908Z 2026-06-24 12:50:10.926 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-24T12:50:10.936857132Z 2026-06-24 12:50:10.936 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-24T12:50:10.941923572Z 2026-06-24 12:50:10.941 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-24T12:50:10.945492757Z 2026-06-24 12:50:10.945 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-24T12:50:10.945581986Z 2026-06-24 12:50:10.945 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-24T12:50:10.979699116Z 2026-06-24 12:50:10.979 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-24T12:50:10.988639547Z 2026-06-24 12:50:10.988 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-24T12:50:11.010570532Z 2026-06-24 12:50:11.009 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-24T12:50:11.024181702Z 2026-06-24 12:50:11.023 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-24T12:50:11.036625746Z 2026-06-24 12:50:11.036 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-24T12:50:11.045645708Z 2026-06-24 12:50:11.045 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-24T12:50:11.112231863Z 2026-06-24 12:50:11.111 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-24T12:50:11.417384430Z 2026-06-24 12:50:11.416 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-24T12:50:11.610396033Z 2026-06-24 12:50:11.609 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-24T12:50:11.899617725Z 2026-06-24 12:50:11.899 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-24T12:50:12.050244425Z 2026-06-24 12:50:12.049 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-24T12:50:12.061863363Z 2026-06-24 12:50:12.061 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_24.request.log
2026-06-24T12:50:12.070501110Z 2026-06-24 12:50:12.068 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-24T12:50:12.071601184Z 2026-06-24 12:50:12.071 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4168ms
2026-06-24T12:50:12.263769104Z 2026-06-24 12:50:12.263 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=48
2026-06-24T12:50:12.336911990Z 2026-06-24 12:50:12.336 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-06-24T12:50:15.856041635Z 2026-06-24 12:50:15.855 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-24T12:50:15.919031551Z 2026-06-24 12:50:15.918 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-24T12:50:15.978123348Z 2026-06-24 12:50:15.977 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-24T12:50:16.305644093Z 2026-06-24 12:50:16.305 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-24T12:50:16.305673184Z 2026-06-24 12:50:16.305 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-24T12:50:16.324276675Z 2026-06-24 12:50:16.323 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-24T12:50:16.746145182Z 2026-06-24 12:50:16.745 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-24T12:50:16.766028981Z 2026-06-24 12:50:16.765 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-24T12:50:16.769632655Z 2026-06-24 12:50:16.769 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-24T12:50:16.770950503Z 2026-06-24 12:50:16.770 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-24T12:50:16.771010431Z 2026-06-24 12:50:16.770 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-24T12:50:16.815531238Z 2026-06-24 12:50:16.814 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-24T12:50:16.818708310Z 2026-06-24 12:50:16.818 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-24T12:50:16.821214867Z 2026-06-24 12:50:16.820 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-24T12:50:16.823387151Z 2026-06-24 12:50:16.823 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-24T12:50:16.834673612Z 2026-06-24 12:50:16.834 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-24T12:50:16.842197339Z 2026-06-24 12:50:16.841 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-24T12:50:16.848360997Z 2026-06-24 12:50:16.847 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-24T12:50:16.848392109Z 2026-06-24 12:50:16.848 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-24T12:50:16.856537937Z 2026-06-24 12:50:16.856 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-24T12:50:16.866514832Z 2026-06-24 12:50:16.865 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-24T12:50:16.867858276Z 2026-06-24 12:50:16.867 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-24T12:50:16.883418414Z 2026-06-24 12:50:16.882 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-24T12:50:16.883448291Z 2026-06-24 12:50:16.883 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-24T12:50:16.883545269Z 2026-06-24 12:50:16.883 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-24T12:50:16.885408802Z 2026-06-24 12:50:16.885 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-24T12:50:16.886098924Z 2026-06-24 12:50:16.885 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-24T12:50:16.887231265Z 2026-06-24 12:50:16.886 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72177986523693060-core_node6-n_0000000000
2026-06-24T12:50:16.895854250Z 2026-06-24 12:50:16.895 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-24T12:50:17.003866540Z 2026-06-24 12:50:17.003 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 205] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:17.008034675Z 2026-06-24 12:50:17.007 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-24T12:50:17.019668861Z 2026-06-24 12:50:17.019 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1165
2026-06-24T12:50:17.119782485Z 2026-06-24 12:50:17.119 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 208] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:17.666532674Z 2026-06-24 12:50:17.666 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:17.779768515Z 2026-06-24 12:50:17.779 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 226] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:18.014365747Z 2026-06-24 12:50:18.013 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:18.126318333Z 2026-06-24 12:50:18.125 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2567981,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-24T12:50:07.669009526Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-24T12:50:08.129493825Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-24T12:50:08.129530239Z It should be set to 65000 to avoid operational disruption.
2026-06-24T12:50:08.129533266Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-24T12:50:08.129920574Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-24T12:50:08.129935422Z
2026-06-24T12:50:08.140139593Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-24T12:50:08.147663856Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-24T12:50:08.193686494Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-24T12:50:08.226834353Z WARNING: A command line option has enabled the Security Manager
2026-06-24T12:50:08.226859177Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-24T12:50:09.348768398Z 2026-06-24 12:50:09.345 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-24T12:50:09.652713595Z 2026-06-24 12:50:09.652 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-24T12:50:09.683463283Z 2026-06-24 12:50:09.682 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-24T12:50:09.691785872Z 2026-06-24 12:50:09.690 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-24T12:50:09.692117224Z 2026-06-24 12:50:09.690 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-24T12:50:09.692534998Z 2026-06-24 12:50:09.691 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-24T12:50:09.692620147Z 2026-06-24 12:50:09.691 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-24T12:50:09.691174886Z
2026-06-24T12:50:09.695331181Z 2026-06-24 12:50:09.694 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-24T12:50:09.720173091Z 2026-06-24 12:50:09.719 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-24T12:50:09.745497671Z 2026-06-24 12:50:09.744 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-24T12:50:09.897645336Z 2026-06-24 12:50:09.897 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-24T12:50:09.919786886Z 2026-06-24 12:50:09.918 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-24T12:50:09.920009513Z 2026-06-24 12:50:09.919 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-24T12:50:09.920267731Z 2026-06-24 12:50:09.919 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-24T12:50:10.031351518Z 2026-06-24 12:50:10.030 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-24T12:50:10.031475886Z 2026-06-24 12:50:10.031 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-24T12:50:10.031616385Z 2026-06-24 12:50:10.031 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-24T12:50:10.070575462Z 2026-06-24 12:50:10.069 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-24T12:50:10.704180541Z 2026-06-24 12:50:10.703 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-24T12:50:10.842468751Z 2026-06-24 12:50:10.841 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-24T12:50:10.844140200Z 2026-06-24 12:50:10.841 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-24T12:50:10.852257493Z 2026-06-24 12:50:10.851 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-24T12:50:10.857071156Z 2026-06-24 12:50:10.856 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-24T12:50:10.861008003Z 2026-06-24 12:50:10.860 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-24T12:50:10.861025728Z 2026-06-24 12:50:10.860 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-24T12:50:10.896481079Z 2026-06-24 12:50:10.895 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-24T12:50:10.905600711Z 2026-06-24 12:50:10.905 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-24T12:50:10.929476394Z 2026-06-24 12:50:10.928 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-24T12:50:10.941191827Z 2026-06-24 12:50:10.940 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-24T12:50:10.955137151Z 2026-06-24 12:50:10.954 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-24T12:50:10.965551679Z 2026-06-24 12:50:10.965 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-24T12:50:11.007764507Z 2026-06-24 12:50:11.007 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-24T12:50:11.045170104Z 2026-06-24 12:50:11.044 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-24T12:50:11.283626494Z 2026-06-24 12:50:11.282 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-24T12:50:11.449686022Z 2026-06-24 12:50:11.449 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-24T12:50:11.707764133Z 2026-06-24 12:50:11.707 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-24T12:50:11.836143653Z 2026-06-24 12:50:11.835 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-24T12:50:11.841603214Z 2026-06-24 12:50:11.841 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_24.request.log
2026-06-24T12:50:11.845324051Z 2026-06-24 12:50:11.844 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-24T12:50:11.846189873Z 2026-06-24 12:50:11.845 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3699ms
2026-06-24T12:50:13.826053778Z 2026-06-24 12:50:13.824 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=62
2026-06-24T12:50:13.928421768Z 2026-06-24 12:50:13.927 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=55
2026-06-24T12:50:15.857850029Z 2026-06-24 12:50:15.857 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-24T12:50:15.933038898Z 2026-06-24 12:50:15.932 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-24T12:50:16.132689128Z 2026-06-24 12:50:16.132 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-24T12:50:16.530869779Z 2026-06-24 12:50:16.530 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-24T12:50:16.530967373Z 2026-06-24 12:50:16.530 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-24T12:50:16.555352367Z 2026-06-24 12:50:16.554 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-06-24T12:50:17.289486480Z 2026-06-24 12:50:17.288 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-24T12:50:17.315404762Z 2026-06-24 12:50:17.314 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-24T12:50:17.318968434Z 2026-06-24 12:50:17.318 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-24T12:50:17.320038391Z 2026-06-24 12:50:17.319 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-24T12:50:17.320053502Z 2026-06-24 12:50:17.319 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-24T12:50:17.477903914Z 2026-06-24 12:50:17.477 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-24T12:50:17.480915587Z 2026-06-24 12:50:17.480 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-24T12:50:17.484227447Z 2026-06-24 12:50:17.483 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-24T12:50:17.488192156Z 2026-06-24 12:50:17.487 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-24T12:50:17.500253582Z 2026-06-24 12:50:17.499 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-24T12:50:17.507566309Z 2026-06-24 12:50:17.507 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-24T12:50:17.514381472Z 2026-06-24 12:50:17.513 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-24T12:50:17.514440245Z 2026-06-24 12:50:17.514 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-24T12:50:17.521184375Z 2026-06-24 12:50:17.520 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-24T12:50:17.533419775Z 2026-06-24 12:50:17.532 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-24T12:50:17.534828168Z 2026-06-24 12:50:17.534 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-24T12:50:17.548903080Z 2026-06-24 12:50:17.548 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-24T12:50:17.548970347Z 2026-06-24 12:50:17.548 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-24T12:50:17.548975379Z 2026-06-24 12:50:17.548 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-06-24T12:50:17.550264191Z 2026-06-24 12:50:17.550 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-24T12:50:17.550833014Z 2026-06-24 12:50:17.550 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-06-24T12:50:17.552059662Z 2026-06-24 12:50:17.551 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72177986523693059-core_node5-n_0000000000
2026-06-24T12:50:17.560002639Z 2026-06-24 12:50:17.559 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-06-24T12:50:17.666697050Z 2026-06-24 12:50:17.666 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:17.670996055Z 2026-06-24 12:50:17.670 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-24T12:50:17.679533338Z 2026-06-24 12:50:17.679 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1823
2026-06-24T12:50:17.780397097Z 2026-06-24 12:50:17.779 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 226] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:18.013935281Z 2026-06-24 12:50:18.013 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-24T12:50:18.126305518Z 2026-06-24 12:50:18.125 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2568160,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-24T12:50:07.880351272Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 30307 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Download complete
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Pulling fs layer
# 9d5a0b6cb36c: Pulling fs layer
# 49518c28c0b3: Pulling fs layer
# 41162acdcc29: Pulling fs layer
# 2ed0fd558ca3: Pulling fs layer
# 2488415fda1c: Pulling fs layer
# 09511c8fb203: Pulling fs layer
# a0c57fb89c4f: Pulling fs layer
# d0641dcc41d7: Pulling fs layer
# 315722fd9570: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 41162acdcc29: Waiting
# 2ed0fd558ca3: Waiting
# 2488415fda1c: Waiting
# 315722fd9570: Waiting
# 4f4fb700ef54: Waiting
# 09511c8fb203: Waiting
# d0641dcc41d7: Waiting
# a0c57fb89c4f: Waiting
# 9d5a0b6cb36c: Verifying Checksum
# 9d5a0b6cb36c: Download complete
# 41162acdcc29: Verifying Checksum
# 41162acdcc29: Download complete
# 2ed0fd558ca3: Verifying Checksum
# 2ed0fd558ca3: Download complete
# 40d16f30db40: Verifying Checksum
# 40d16f30db40: Download complete
# 09511c8fb203: Verifying Checksum
# 09511c8fb203: Download complete
# 49518c28c0b3: Verifying Checksum
# 49518c28c0b3: Download complete
# a0c57fb89c4f: Verifying Checksum
# a0c57fb89c4f: Download complete
# d0641dcc41d7: Verifying Checksum
# d0641dcc41d7: Download complete
# 315722fd9570: Verifying Checksum
# 315722fd9570: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 2488415fda1c: Verifying Checksum
# 2488415fda1c: Download complete
# 40d16f30db40: Pull complete
# 9d5a0b6cb36c: Pull complete
# 49518c28c0b3: Pull complete
# 41162acdcc29: Pull complete
# 2ed0fd558ca3: Pull complete
# 2488415fda1c: Pull complete
# 09511c8fb203: Pull complete
# a0c57fb89c4f: Pull complete
# d0641dcc41d7: Pull complete
# 315722fd9570: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# d0cca0692e3d313d6251a5fb6992d83eee62e45a97f5f4c5b2d2d51bee849740
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 8b2b0c2f23482326f6352bd96a17858d2ee429f6313a8dec5fdaeaa8df9e933c
# Starting solr-node2 connected to first node's ZooKeeper
# 30e0ee109ba7cc8fbd3c7156b6bc80dc2e36a84a95afc0838d2dbc53526acc4c
# Started solr-node3 connected to first node's ZooKeeper
# a5f9fdebcff4c2ced8675cb9310de7f7f4d586ddafc2e7b73d704062e6022d9f
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-24 12:50:15.157; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-24 12:50:15.189; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 16420 ms
ok 87 snapshot lifecycle # in 5272 ms
ok 88 snapshot list # in 3513 ms
ok 89 snapshot list via --solr-connection # in 3674 ms
ok 90 snapshot describe # in 3573 ms
ok 91 snapshot describe via --solr-connection # in 3666 ms
ok 92 start solr with ssl # in 17626 ms
ok 93 use different hostname when not checking peer-name # in 26492 ms
ok 94 start solr with ssl and auth # in 15116 ms
ok 95 start solr with client truststore and security manager # in 17882 ms
ok 96 start solr with mTLS needed # in 49057 ms
ok 97 start solr with mTLS wanted # in 36457 ms
ok 98 test keystore reload # in 61694 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 16921 ms
ok 100 stop command for single port # in 19227 ms
ok 101 check stop command doesn't hang # in 227 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8887 ms
not ok 103 deprecated system properties converted to modern properties # in 195478 ms
# (from function `assert_file_contains' in file .gradle/node/node_modules/bats-file/src/file.bash, line 566,
# in test file test/test_start_solr.bats, line 86)
# `assert_file_contains "${SOLR_LOGS_DIR}/solr.log" 'You are passing in deprecated system property disable.config.edit and should upgrade to using solr.api.config.edit.enabled instead.'' failed
# Waiting up to 180 seconds to see Solr running on port 40685 [|] [/] [-] [\] [|]
# Started Solr server on port 40685 (pid=2585689). Happy searching!
#
#
# -- file does not contain regex --
# path : /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/solr.log
# --
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-5
ok 104 start with custom jetty options # in 8537 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 10112 ms
not ok 106 bootstrapping a configset # in 9868 ms
# (from function `config_exists' in file test/bats_helper.bash, line 84,
# in test file test/test_start_solr.bats, line 116)
# `config_exists "techproducts"' failed
# Waiting up to 180 seconds to see Solr running on port 40685 [|] [/] [-] [\] [|]
# Started Solr server on port 40685 (pid=2588401). Happy searching!
#
# WARN - 2026-06-24 12:59:16.847; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-24 12:59:16.851; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-8
ok 107 status detects locally running solr # in 10381 ms
ok 108 status with --solr-url from user # in 8643 ms
ok 109 status with --port from user # in 8480 ms
ok 110 multiple connection options are prevented # in 949 ms
ok 111 status with invalid --solr-url from user # in 1376 ms
ok 112 status with --short format # in 196535 ms
ok 113 searching solr via locally executed streaming expression # in 1928 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1669 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1268 ms
ok 116 variable interpolation # in 1811 ms
ok 117 searching solr without credentials fails with error # in 1238 ms
ok 118 --version returns Solr version # in 786 ms
ok 119 version as direct tool call still runs # in 782 ms
ok 120 short help # in 931 ms
ok 121 short help is inferred # in 811 ms
ok 122 long help # in 912 ms
ok 123 running subcommands with zk is prevented # in 834 ms
ok 124 listing out files # in 2215 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 10151 ms
ok 126 copying files around # in 11822 ms
ok 127 upconfig # in 2790 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1435 ms
ok 129 downconfig # in 1434 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1569 ms
ok 131 env var ZK_HOST is honored # in 3453 ms
ok 132 Cleanup # in 1922 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 39s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/n22mbsns353xy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 242 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/242/
No tests ran.
Build Log:
[...truncated 902 lines...]
BUILD FAILED in 30m 56s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/y7yc6fpigol3o
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 241 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/241/
No tests ran.
Build Log:
[...truncated 903 lines...]
BUILD FAILED in 29m
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/fi2zajm4onmxg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 240 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/240/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-21 12:04:28.517; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-21 12:04:28.520; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-21 12:04:28.966; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-21 12:04:28.966; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 16261 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 44801 is already being used by another process (pid: 3253843)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-21 12:04:30.081; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-21 12:04:30.084; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 10 post, postlogs and export with basic auth # in 17243 ms
ok 11 package with basic auth # in 11696 ms
ok 12 admin UI static content is excepted from basic auth # in 10143 ms
ok 13 nothing # in 9 ms
ok 14 server does not compress response without Accept-Encoding header # in 137 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 76 ms
ok 16 compressed response can be decompressed and parsed # in 77 ms
ok 17 setting property # in 10377 ms
ok 18 short form of setting property # in 6235 ms
ok 19 connecting to solr via various connection types # in 10040 ms
ok 20 create for non cloud mode # in 7551 ms
ok 21 create for cloud mode # in 9600 ms
ok 22 multiple connection options are prevented # in 792 ms
ok 23 create collection # in 7154 ms
ok 24 create collection using solrUrl # in 4932 ms
ok 25 create collection using legacy solrUrl # in 4928 ms
ok 26 create collection using Zookeeper # in 5092 ms
ok 27 reject d option with invalid config dir # in 2684 ms
ok 28 accept d option with builtin config # in 4904 ms
ok 29 accept d option with explicit path to config # in 5077 ms
ok 30 accept n option as config name # in 4927 ms
ok 31 allow config reuse when n option specifies same config # in 8728 ms
ok 32 create multisharded collections when s provided # in 4889 ms
ok 33 create replicated collections when rf provided # in 5819 ms
ok 34 can delete collections # in 10019 ms
ok 35 can delete collections with solr-url # in 7850 ms
ok 36 can delete collections with solr-connection # in 7733 ms
ok 37 collection delete also deletes zk config # in 7070 ms
ok 38 deletes accompanying zk config with nondefault name # in 7088 ms
ok 39 delete-config option can opt to leave config in zk # in 6780 ms
ok 40 start -e cloud works with --jvm-opts # in 20931 ms
ok 41 start -e cloud works with --no-prompt # in 46186 ms
ok 42 Check export command # in 18071 ms
ok 43 export fails on non cloud mode # in 9039 ms
Waiting for Tika Server to be ready on port 44806
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16541 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16504 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16500 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 9861 ms
ok 48 healthcheck on remote cloud solr # in 9393 ms
ok 49 healthcheck errors on standalone solr # in 10625 ms
ok 50 solr help flag prints help # in 695 ms
ok 51 solr with no flags prints help # in 695 ms
ok 52 start help flag prints help # in 137 ms
ok 53 start h flag prints help # in 142 ms
ok 54 stop help flag prints help # in 138 ms
ok 55 restart help flag prints help # in 138 ms
ok 56 status help flag prints help # in 737 ms
ok 57 healthcheck help flag prints help # in 702 ms
ok 58 create help flag prints help # in 718 ms
ok 59 delete help flag prints help # in 715 ms
ok 60 zk help flag prints help # in 720 ms
ok 61 auth help flag prints help # in 725 ms
ok 62 assert help flag prints help # in 706 ms
ok 63 post help flag prints help # in 728 ms
ok 64 SQL Module # in 14980 ms
ok 65 icu collation in analysis-extras module # in 11859 ms
ok 66 Check lifecycle of sentiment classification # in 25992 ms
ok 67 lifecycle of package # in 8143 ms
ok 68 deploying and undeploying a collection level package # in 12641 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 14916 ms
ok 71 Random placement plugin using ENV # in 14783 ms
ok 72 Check help command # in 2210 ms
ok 73 basic post with a type specified # in 5072 ms
ok 74 basic post with collection instead of url parameter # in 3830 ms
ok 75 basic post with solr-url and collection # in 3733 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 3691 ms
ok 77 basic post WITHOUT a type specified # in 10756 ms
ok 78 crawling a directory as a dry-run # in 787 ms
ok 79 crawling a directory # in 4235 ms
ok 80 skipcommit and optimize and delete # in 3652 ms
ok 81 args mode # in 6591 ms
ok 82 stdin mode # in 3189 ms
ok 83 verbose echo the Solr response # in 3656 ms
ok 84 post solr log into solr via cli # in 8907 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-21T12:14:52.467128068Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-21T12:14:52.467189998Z It should be set to 65000 to avoid operational disruption.
2026-06-21T12:14:52.467193736Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-21T12:14:52.467435365Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-21T12:14:52.467446772Z
2026-06-21T12:14:52.476537828Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-21T12:14:52.484133871Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-21T12:14:52.526901975Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-21T12:14:52.558083148Z WARNING: A command line option has enabled the Security Manager
2026-06-21T12:14:52.558121883Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-21T12:14:53.597934808Z 2026-06-21 12:14:53.594 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-21T12:14:53.855872460Z 2026-06-21 12:14:53.855 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-21T12:14:53.879642099Z 2026-06-21 12:14:53.879 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-21T12:14:53.886572878Z 2026-06-21 12:14:53.886 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-21T12:14:53.886638739Z 2026-06-21 12:14:53.886 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-21T12:14:53.886758010Z 2026-06-21 12:14:53.886 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-21T12:14:53.886895159Z 2026-06-21 12:14:53.886 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-21T12:14:53.886311774Z
2026-06-21T12:14:53.889308019Z 2026-06-21 12:14:53.888 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-21T12:14:53.910599279Z 2026-06-21 12:14:53.910 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-21T12:14:53.923256629Z 2026-06-21 12:14:53.922 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-21T12:14:53.923318491Z 2026-06-21 12:14:53.922 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-21T12:14:53.956077726Z 2026-06-21 12:14:53.955 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-21T12:14:54.608742136Z 2026-06-21 12:14:54.608 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-21T12:14:54.824738791Z 2026-06-21 12:14:54.824 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-21T12:14:54.829015336Z 2026-06-21 12:14:54.828 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-21T12:14:54.829028165Z 2026-06-21 12:14:54.828 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-21T12:14:54.831120146Z 2026-06-21 12:14:54.830 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-21T12:14:54.831228217Z 2026-06-21 12:14:54.830 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-21T12:14:54.864850947Z 2026-06-21 12:14:54.864 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-21T12:14:55.334618416Z 2026-06-21 12:14:55.333 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-21T12:14:55.347431449Z 2026-06-21 12:14:55.346 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-21T12:14:55.384403353Z 2026-06-21 12:14:55.383 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-21T12:14:55.427368831Z 2026-06-21 12:14:55.424 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-21T12:14:55.427446378Z 2026-06-21 12:14:55.425 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-21T12:14:55.616838271Z 2026-06-21 12:14:55.615 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-21T12:14:55.638846679Z 2026-06-21 12:14:55.638 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-21T12:14:55.654143816Z 2026-06-21 12:14:55.653 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-21T12:14:55.660357329Z 2026-06-21 12:14:55.660 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72160861276012544-solr-node1:8983_solr-n_0000000000) starting
2026-06-21T12:14:55.721844232Z 2026-06-21 12:14:55.721 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-21T12:14:55.723191329Z 2026-06-21 12:14:55.722 INFO (OverseerStateUpdate-72160861276012544-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-21T12:14:55.731628919Z 2026-06-21 12:14:55.731 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-21T12:14:55.777598909Z 2026-06-21 12:14:55.777 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-21T12:14:56.204558278Z 2026-06-21 12:14:56.203 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-21T12:14:56.397436814Z 2026-06-21 12:14:56.396 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-21T12:14:56.648350489Z 2026-06-21 12:14:56.647 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-21T12:14:56.773178877Z 2026-06-21 12:14:56.772 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-21T12:14:56.778651866Z 2026-06-21 12:14:56.778 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_21.request.log
2026-06-21T12:14:56.782068109Z 2026-06-21 12:14:56.781 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-21T12:14:56.783692508Z 2026-06-21 12:14:56.783 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4300ms
2026-06-21T12:14:56.968020104Z 2026-06-21 12:14:56.967 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-06-21T12:14:57.030357820Z 2026-06-21 12:14:57.029 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-06-21T12:15:00.174668403Z 2026-06-21 12:15:00.174 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-21T12:15:00.203057669Z 2026-06-21 12:15:00.202 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-21T12:15:04.074894989Z 2026-06-21 12:15:04.074 INFO (qtp985324122-100-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-06-21T12:15:04.099059028Z 2026-06-21 12:15:04.098 INFO (qtp985324122-100-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=7
2026-06-21T12:15:04.124150726Z 2026-06-21 12:15:04.123 INFO (qtp985324122-101-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1
2026-06-21T12:15:04.610488646Z 2026-06-21 12:15:04.609 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=4
2026-06-21T12:15:04.637143278Z 2026-06-21 12:15:04.634 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-21T12:15:04.651767540Z 2026-06-21 12:15:04.651 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-21T12:15:04.656301794Z 2026-06-21 12:15:04.655 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:04.691364791Z 2026-06-21 12:15:04.690 INFO (OverseerStateUpdate-72160861276012544-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-21T12:15:04.691381507Z "core":"test-collection_shard2_replica_n1",
2026-06-21T12:15:04.691384222Z "node_name":"solr-node3:8985_solr",
2026-06-21T12:15:04.691386494Z "base_url":"http://solr-node3:8985/solr",
2026-06-21T12:15:04.691388764Z "collection":"test-collection",
2026-06-21T12:15:04.691390809Z "shard":"shard2",
2026-06-21T12:15:04.691392765Z "state":"down",
2026-06-21T12:15:04.691394717Z "type":"NRT",
2026-06-21T12:15:04.691396600Z "operation":"ADDREPLICA",
2026-06-21T12:15:04.691399050Z "waitForFinalState":"false"}
2026-06-21T12:15:04.697257900Z 2026-06-21 12:15:04.696 INFO (OverseerStateUpdate-72160861276012544-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-21T12:15:04.697266334Z "core":"test-collection_shard3_replica_n2",
2026-06-21T12:15:04.697268717Z "node_name":"solr-node1:8983_solr",
2026-06-21T12:15:04.697270663Z "base_url":"http://solr-node1:8983/solr",
2026-06-21T12:15:04.697272740Z "collection":"test-collection",
2026-06-21T12:15:04.697274818Z "shard":"shard3",
2026-06-21T12:15:04.697276773Z "state":"down",
2026-06-21T12:15:04.697278658Z "type":"NRT",
2026-06-21T12:15:04.697280634Z "operation":"ADDREPLICA",
2026-06-21T12:15:04.697282669Z "waitForFinalState":"false"}
2026-06-21T12:15:04.703114610Z 2026-06-21 12:15:04.702 INFO (OverseerStateUpdate-72160861276012544-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-21T12:15:04.703139957Z "core":"test-collection_shard1_replica_n4",
2026-06-21T12:15:04.703143562Z "node_name":"solr-node2:8984_solr",
2026-06-21T12:15:04.703146527Z "base_url":"http://solr-node2:8984/solr",
2026-06-21T12:15:04.703149313Z "collection":"test-collection",
2026-06-21T12:15:04.703151980Z "shard":"shard1",
2026-06-21T12:15:04.703154757Z "state":"down",
2026-06-21T12:15:04.703157582Z "type":"NRT",
2026-06-21T12:15:04.703161054Z "operation":"ADDREPLICA",
2026-06-21T12:15:04.703163787Z "waitForFinalState":"false"}
2026-06-21T12:15:04.813036112Z 2026-06-21 12:15:04.812 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:04.871533162Z 2026-06-21 12:15:04.871 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard3_replica_n2&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-21T12:15:04.924169860Z 2026-06-21 12:15:04.923 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-21T12:15:04.976384647Z 2026-06-21 12:15:04.975 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-21T12:15:05.252660591Z 2026-06-21 12:15:05.249 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-21T12:15:05.253280385Z 2026-06-21 12:15:05.249 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-21T12:15:05.279271733Z 2026-06-21 12:15:05.278 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n2], dataDir=[/var/solr/data/test-collection_shard3_replica_n2/data/]
2026-06-21T12:15:05.947188950Z 2026-06-21 12:15:05.944 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-21T12:15:06.000548924Z 2026-06-21 12:15:05.999 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-21T12:15:06.004704402Z 2026-06-21 12:15:06.004 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-21T12:15:06.007331862Z 2026-06-21 12:15:06.006 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-21T12:15:06.007720571Z 2026-06-21 12:15:06.007 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-21T12:15:06.075316181Z 2026-06-21 12:15:06.074 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-21T12:15:06.077602004Z 2026-06-21 12:15:06.077 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-21T12:15:06.079933330Z 2026-06-21 12:15:06.079 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-21T12:15:06.081859304Z 2026-06-21 12:15:06.081 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-21T12:15:06.098539733Z 2026-06-21 12:15:06.098 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-21T12:15:06.107196012Z 2026-06-21 12:15:06.106 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-21T12:15:06.113622555Z 2026-06-21 12:15:06.113 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n2 solr-node1-8 core_node5 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-21T12:15:06.113644451Z 2026-06-21 12:15:06.113 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n2 solr-node1-8 core_node5 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-21T12:15:06.122980394Z 2026-06-21 12:15:06.122 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n2 solr-node1-8 core_node5 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-21T12:15:06.128658176Z 2026-06-21 12:15:06.128 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-21T12:15:06.130288325Z 2026-06-21 12:15:06.129 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-21T12:15:06.146058441Z 2026-06-21 12:15:06.145 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-21T12:15:06.146144691Z 2026-06-21 12:15:06.145 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-21T12:15:06.146250319Z 2026-06-21 12:15:06.145 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n2/
2026-06-21T12:15:06.147367431Z 2026-06-21 12:15:06.147 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-21T12:15:06.147910755Z 2026-06-21 12:15:06.147 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n2/ has no replicas
2026-06-21T12:15:06.149535191Z 2026-06-21 12:15:06.149 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72160861276012544-core_node5-n_0000000000
2026-06-21T12:15:06.157246621Z 2026-06-21 12:15:06.156 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n2/ shard3
2026-06-21T12:15:06.321887379Z 2026-06-21 12:15:06.321 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:06.326321074Z 2026-06-21 12:15:06.325 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-21T12:15:06.336180183Z 2026-06-21 12:15:06.335 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1465
2026-06-21T12:15:06.350362604Z 2026-06-21 12:15:06.350 INFO (qtp985324122-27-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-21T12:15:06.436832288Z 2026-06-21 12:15:06.436 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:06.439193505Z 2026-06-21 12:15:06.438 INFO (qtp985324122-27-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=1819
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3285844,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-21T12:14:52.25531223Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-21T12:14:57.353203172Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-21T12:14:57.353239324Z It should be set to 65000 to avoid operational disruption.
2026-06-21T12:14:57.353243340Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-21T12:14:57.353742386Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-21T12:14:57.353751581Z
2026-06-21T12:14:57.365333402Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-21T12:14:57.373971950Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-21T12:14:57.419774543Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-21T12:14:57.459734406Z WARNING: A command line option has enabled the Security Manager
2026-06-21T12:14:57.459753624Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-21T12:14:58.575764869Z 2026-06-21 12:14:58.572 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-21T12:14:58.841441519Z 2026-06-21 12:14:58.840 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-21T12:14:58.867026472Z 2026-06-21 12:14:58.866 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-21T12:14:58.874144295Z 2026-06-21 12:14:58.873 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-21T12:14:58.874236982Z 2026-06-21 12:14:58.873 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-21T12:14:58.874334582Z 2026-06-21 12:14:58.873 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-21T12:14:58.874433810Z 2026-06-21 12:14:58.873 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-21T12:14:58.873953314Z
2026-06-21T12:14:58.877032937Z 2026-06-21 12:14:58.876 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-21T12:14:58.899107601Z 2026-06-21 12:14:58.898 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-21T12:14:58.921856941Z 2026-06-21 12:14:58.921 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-21T12:14:59.058791005Z 2026-06-21 12:14:59.058 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-21T12:14:59.075983499Z 2026-06-21 12:14:59.075 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-21T12:14:59.076070281Z 2026-06-21 12:14:59.075 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-21T12:14:59.076176644Z 2026-06-21 12:14:59.075 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-21T12:14:59.185915611Z 2026-06-21 12:14:59.185 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-21T12:14:59.185980625Z 2026-06-21 12:14:59.185 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-21T12:14:59.186094019Z 2026-06-21 12:14:59.185 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-21T12:14:59.217870211Z 2026-06-21 12:14:59.217 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-21T12:14:59.900497465Z 2026-06-21 12:14:59.899 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-21T12:15:00.052972126Z 2026-06-21 12:15:00.052 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-21T12:15:00.054443105Z 2026-06-21 12:15:00.052 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-21T12:15:00.063711987Z 2026-06-21 12:15:00.063 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-21T12:15:00.069481108Z 2026-06-21 12:15:00.069 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-21T12:15:00.072483265Z 2026-06-21 12:15:00.072 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-21T12:15:00.072496908Z 2026-06-21 12:15:00.072 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-21T12:15:00.107128297Z 2026-06-21 12:15:00.106 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-21T12:15:00.115846587Z 2026-06-21 12:15:00.115 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-21T12:15:00.141718407Z 2026-06-21 12:15:00.141 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-21T12:15:00.152420956Z 2026-06-21 12:15:00.152 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-21T12:15:00.165285007Z 2026-06-21 12:15:00.164 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-21T12:15:00.175690473Z 2026-06-21 12:15:00.175 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-21T12:15:00.203739143Z 2026-06-21 12:15:00.202 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-21T12:15:00.219676952Z 2026-06-21 12:15:00.219 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-21T12:15:00.522380695Z 2026-06-21 12:15:00.521 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-21T12:15:00.706138585Z 2026-06-21 12:15:00.705 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-21T12:15:00.978477849Z 2026-06-21 12:15:00.977 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-21T12:15:01.118124027Z 2026-06-21 12:15:01.117 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-21T12:15:01.123787744Z 2026-06-21 12:15:01.123 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_21.request.log
2026-06-21T12:15:01.127879717Z 2026-06-21 12:15:01.127 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-21T12:15:01.128755040Z 2026-06-21 12:15:01.128 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3756ms
2026-06-21T12:15:01.285062905Z 2026-06-21 12:15:01.284 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=43
2026-06-21T12:15:01.361205801Z 2026-06-21 12:15:01.358 INFO (qtp985324122-56-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-06-21T12:15:04.871816262Z 2026-06-21 12:15:04.871 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-21T12:15:04.941515468Z 2026-06-21 12:15:04.940 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-21T12:15:05.018405483Z 2026-06-21 12:15:05.017 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-21T12:15:05.282115249Z 2026-06-21 12:15:05.281 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-21T12:15:05.282133541Z 2026-06-21 12:15:05.281 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-21T12:15:05.300064210Z 2026-06-21 12:15:05.299 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-21T12:15:05.711129515Z 2026-06-21 12:15:05.710 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-21T12:15:05.731088557Z 2026-06-21 12:15:05.730 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-21T12:15:05.734944236Z 2026-06-21 12:15:05.734 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-21T12:15:05.736061659Z 2026-06-21 12:15:05.735 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-21T12:15:05.736082046Z 2026-06-21 12:15:05.735 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-21T12:15:05.842314006Z 2026-06-21 12:15:05.841 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-21T12:15:05.845192354Z 2026-06-21 12:15:05.844 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-21T12:15:05.847129625Z 2026-06-21 12:15:05.846 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-21T12:15:05.849093678Z 2026-06-21 12:15:05.848 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-21T12:15:05.860089671Z 2026-06-21 12:15:05.859 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-21T12:15:05.867482793Z 2026-06-21 12:15:05.867 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-21T12:15:05.872985169Z 2026-06-21 12:15:05.872 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-21T12:15:05.873089265Z 2026-06-21 12:15:05.872 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-21T12:15:05.879453232Z 2026-06-21 12:15:05.879 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-21T12:15:05.892295805Z 2026-06-21 12:15:05.891 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-21T12:15:05.893755422Z 2026-06-21 12:15:05.893 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-21T12:15:05.907920158Z 2026-06-21 12:15:05.907 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-21T12:15:05.907940871Z 2026-06-21 12:15:05.907 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-21T12:15:05.907997397Z 2026-06-21 12:15:05.907 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-21T12:15:05.909165201Z 2026-06-21 12:15:05.908 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-21T12:15:05.909872741Z 2026-06-21 12:15:05.909 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-21T12:15:05.910857809Z 2026-06-21 12:15:05.910 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72160861276012547-core_node6-n_0000000000
2026-06-21T12:15:05.919856805Z 2026-06-21 12:15:05.919 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-21T12:15:06.030574387Z 2026-06-21 12:15:06.030 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 205] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:06.035093781Z 2026-06-21 12:15:06.034 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-21T12:15:06.044762509Z 2026-06-21 12:15:06.044 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1174
2026-06-21T12:15:06.143102550Z 2026-06-21 12:15:06.142 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:06.321174986Z 2026-06-21 12:15:06.320 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:06.437055297Z 2026-06-21 12:15:06.436 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3286243,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-21T12:14:57.164387683Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-21T12:14:57.546418945Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-21T12:14:57.546460796Z It should be set to 65000 to avoid operational disruption.
2026-06-21T12:14:57.546463822Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-21T12:14:57.546710228Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-21T12:14:57.546716924Z
2026-06-21T12:14:57.556546266Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-21T12:14:57.564209985Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-21T12:14:57.606787455Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-21T12:14:57.636923125Z WARNING: A command line option has enabled the Security Manager
2026-06-21T12:14:57.636947229Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-21T12:14:58.639048029Z 2026-06-21 12:14:58.635 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-21T12:14:58.902818962Z 2026-06-21 12:14:58.902 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-21T12:14:58.927533320Z 2026-06-21 12:14:58.927 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-21T12:14:58.934445592Z 2026-06-21 12:14:58.933 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-21T12:14:58.934520205Z 2026-06-21 12:14:58.934 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-21T12:14:58.934620445Z 2026-06-21 12:14:58.934 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-21T12:14:58.934765328Z 2026-06-21 12:14:58.934 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-21T12:14:58.934142400Z
2026-06-21T12:14:58.937481332Z 2026-06-21 12:14:58.937 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-21T12:14:58.959172784Z 2026-06-21 12:14:58.958 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-21T12:14:58.981929833Z 2026-06-21 12:14:58.981 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-21T12:14:59.118901024Z 2026-06-21 12:14:59.118 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-21T12:14:59.135865147Z 2026-06-21 12:14:59.135 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-21T12:14:59.135990091Z 2026-06-21 12:14:59.135 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-21T12:14:59.136123894Z 2026-06-21 12:14:59.135 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-21T12:14:59.244288005Z 2026-06-21 12:14:59.243 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-21T12:14:59.244502389Z 2026-06-21 12:14:59.244 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-21T12:14:59.244512876Z 2026-06-21 12:14:59.244 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-21T12:14:59.276792540Z 2026-06-21 12:14:59.276 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-21T12:14:59.954590087Z 2026-06-21 12:14:59.954 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-21T12:15:00.097475828Z 2026-06-21 12:15:00.095 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-21T12:15:00.100599388Z 2026-06-21 12:15:00.095 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-21T12:15:00.108180014Z 2026-06-21 12:15:00.107 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-21T12:15:00.113031944Z 2026-06-21 12:15:00.112 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-21T12:15:00.116096115Z 2026-06-21 12:15:00.115 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-21T12:15:00.116108756Z 2026-06-21 12:15:00.115 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-21T12:15:00.145477934Z 2026-06-21 12:15:00.144 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-21T12:15:00.153543671Z 2026-06-21 12:15:00.153 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-21T12:15:00.173576494Z 2026-06-21 12:15:00.173 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-21T12:15:00.175436927Z 2026-06-21 12:15:00.175 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-21T12:15:00.183952203Z 2026-06-21 12:15:00.183 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-21T12:15:00.193853545Z 2026-06-21 12:15:00.193 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-21T12:15:00.204383190Z 2026-06-21 12:15:00.203 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-21T12:15:00.251782885Z 2026-06-21 12:15:00.251 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-21T12:15:00.515770885Z 2026-06-21 12:15:00.515 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-21T12:15:00.683501596Z 2026-06-21 12:15:00.682 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-21T12:15:00.929458536Z 2026-06-21 12:15:00.929 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-21T12:15:01.048293619Z 2026-06-21 12:15:01.047 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-21T12:15:01.053976405Z 2026-06-21 12:15:01.053 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_21.request.log
2026-06-21T12:15:01.057363309Z 2026-06-21 12:15:01.056 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-21T12:15:01.058159068Z 2026-06-21 12:15:01.057 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3495ms
2026-06-21T12:15:02.947584687Z 2026-06-21 12:15:02.946 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=191
2026-06-21T12:15:03.011170632Z 2026-06-21 12:15:03.010 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-06-21T12:15:04.876497183Z 2026-06-21 12:15:04.875 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-21T12:15:04.937141512Z 2026-06-21 12:15:04.936 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-21T12:15:04.993786775Z 2026-06-21 12:15:04.993 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-06-21T12:15:05.261524930Z 2026-06-21 12:15:05.260 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-21T12:15:05.261542848Z 2026-06-21 12:15:05.261 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-21T12:15:05.279862871Z 2026-06-21 12:15:05.279 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n1], dataDir=[/var/solr/data/test-collection_shard2_replica_n1/data/]
2026-06-21T12:15:05.921613714Z 2026-06-21 12:15:05.921 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-21T12:15:05.954347330Z 2026-06-21 12:15:05.953 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-21T12:15:05.968023714Z 2026-06-21 12:15:05.967 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-21T12:15:05.971055459Z 2026-06-21 12:15:05.970 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-21T12:15:05.971323041Z 2026-06-21 12:15:05.970 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-21T12:15:06.062111640Z 2026-06-21 12:15:06.061 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-21T12:15:06.068308886Z 2026-06-21 12:15:06.067 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-21T12:15:06.073209056Z 2026-06-21 12:15:06.072 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-21T12:15:06.077876755Z 2026-06-21 12:15:06.077 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-21T12:15:06.120573110Z 2026-06-21 12:15:06.119 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-21T12:15:06.156200533Z 2026-06-21 12:15:06.155 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-21T12:15:06.167872676Z 2026-06-21 12:15:06.167 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-8 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-21T12:15:06.167890332Z 2026-06-21 12:15:06.167 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-8 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-21T12:15:06.175584491Z 2026-06-21 12:15:06.175 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-8 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-21T12:15:06.185109755Z 2026-06-21 12:15:06.184 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-21T12:15:06.186625278Z 2026-06-21 12:15:06.186 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-21T12:15:06.201557538Z 2026-06-21 12:15:06.201 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-21T12:15:06.201704305Z 2026-06-21 12:15:06.201 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-21T12:15:06.201717980Z 2026-06-21 12:15:06.201 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n1/
2026-06-21T12:15:06.203369219Z 2026-06-21 12:15:06.203 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-21T12:15:06.203993896Z 2026-06-21 12:15:06.203 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ has no replicas
2026-06-21T12:15:06.206316279Z 2026-06-21 12:15:06.206 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72160861276012548-core_node3-n_0000000000
2026-06-21T12:15:06.215129698Z 2026-06-21 12:15:06.214 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ shard2
2026-06-21T12:15:06.321253049Z 2026-06-21 12:15:06.320 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-21T12:15:06.326979336Z 2026-06-21 12:15:06.326 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-21T12:15:06.336621507Z 2026-06-21 12:15:06.336 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1461
2026-06-21T12:15:06.437206024Z 2026-06-21 12:15:06.436 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3286412,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-21T12:14:57.341748162Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 28456 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Pulling fs layer
# 601f2c23751f: Waiting
# 0ed719aef393: Pulling fs layer
# 0ed719aef393: Waiting
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 4f4fb700ef54: Waiting
# 3aa9120e46f5: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Pulling fs layer
# 9d5a0b6cb36c: Pulling fs layer
# 49518c28c0b3: Pulling fs layer
# 41162acdcc29: Pulling fs layer
# 2ed0fd558ca3: Pulling fs layer
# 2488415fda1c: Pulling fs layer
# 09511c8fb203: Pulling fs layer
# a0c57fb89c4f: Pulling fs layer
# d0641dcc41d7: Pulling fs layer
# 315722fd9570: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# a0c57fb89c4f: Waiting
# 2ed0fd558ca3: Waiting
# 2488415fda1c: Waiting
# 09511c8fb203: Waiting
# d0641dcc41d7: Waiting
# 4f4fb700ef54: Waiting
# 315722fd9570: Waiting
# 41162acdcc29: Waiting
# 40d16f30db40: Verifying Checksum
# 40d16f30db40: Download complete
# 9d5a0b6cb36c: Verifying Checksum
# 9d5a0b6cb36c: Download complete
# 41162acdcc29: Verifying Checksum
# 41162acdcc29: Download complete
# 2ed0fd558ca3: Verifying Checksum
# 2ed0fd558ca3: Download complete
# 09511c8fb203: Verifying Checksum
# 09511c8fb203: Download complete
# 49518c28c0b3: Verifying Checksum
# 49518c28c0b3: Download complete
# a0c57fb89c4f: Verifying Checksum
# a0c57fb89c4f: Download complete
# d0641dcc41d7: Download complete
# 315722fd9570: Download complete
# 4f4fb700ef54: Download complete
# 2488415fda1c: Verifying Checksum
# 2488415fda1c: Download complete
# 40d16f30db40: Pull complete
# 9d5a0b6cb36c: Pull complete
# 49518c28c0b3: Pull complete
# 41162acdcc29: Pull complete
# 2ed0fd558ca3: Pull complete
# 2488415fda1c: Pull complete
# 09511c8fb203: Pull complete
# a0c57fb89c4f: Pull complete
# d0641dcc41d7: Pull complete
# 315722fd9570: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 33b7cbc3d312472ced44d510a7477591dae6b7d9818e60e724a9b09df5dc2cf1
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 14fec5eb9f516f6341d0d85d816603768c87e7405d4c038c28aafd0668c64131
# Starting solr-node2 connected to first node's ZooKeeper
# eaf9d1d61dbf4094883e7914c12937526587281a4c8e2a7f8d4c8448a9004cf8
# Started solr-node3 connected to first node's ZooKeeper
# 113060fb372336344de57b66d8706f7e5312635e6001b326ef62fb754e966432
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-21 12:15:04.153; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-21 12:15:04.183; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 14802 ms
ok 87 snapshot lifecycle # in 4559 ms
ok 88 snapshot list # in 3133 ms
ok 89 snapshot list via --solr-connection # in 3139 ms
ok 90 snapshot describe # in 3214 ms
ok 91 snapshot describe via --solr-connection # in 3169 ms
ok 92 start solr with ssl # in 15803 ms
ok 93 use different hostname when not checking peer-name # in 23566 ms
ok 94 start solr with ssl and auth # in 14101 ms
ok 95 start solr with client truststore and security manager # in 15819 ms
ok 96 start solr with mTLS needed # in 44476 ms
ok 97 start solr with mTLS wanted # in 36841 ms
ok 98 test keystore reload # in 56790 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 15472 ms
ok 100 stop command for single port # in 17763 ms
ok 101 check stop command doesn't hang # in 205 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8020 ms
not ok 103 deprecated system properties converted to modern properties # in 195359 ms
# (from function `assert_file_contains' in file .gradle/node/node_modules/bats-file/src/file.bash, line 566,
# in test file test/test_start_solr.bats, line 86)
# `assert_file_contains "${SOLR_LOGS_DIR}/solr.log" 'You are passing in deprecated system property disable.config.edit and should upgrade to using solr.api.config.edit.enabled instead.'' failed
# Waiting up to 180 seconds to see Solr running on port 44801 [|] [/] [-] [\] [|]
# Started Solr server on port 44801 (pid=3303944). Happy searching!
#
#
# -- file does not contain regex --
# path : /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/solr.log
# --
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-5
ok 104 start with custom jetty options # in 7881 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 8917 ms
not ok 106 bootstrapping a configset # in 8782 ms
# (from function `config_exists' in file test/bats_helper.bash, line 84,
# in test file test/test_start_solr.bats, line 116)
# `config_exists "techproducts"' failed
# Waiting up to 180 seconds to see Solr running on port 44801 [|] [/] [-] [\] [|]
# Started Solr server on port 44801 (pid=3306660). Happy searching!
#
# WARN - 2026-06-21 12:23:36.242; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-21 12:23:36.246; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-8
ok 107 status detects locally running solr # in 9168 ms
ok 108 status with --solr-url from user # in 7745 ms
ok 109 status with --port from user # in 7716 ms
ok 110 multiple connection options are prevented # in 809 ms
ok 111 status with invalid --solr-url from user # in 1174 ms
ok 112 status with --short format # in 195594 ms
ok 113 searching solr via locally executed streaming expression # in 1663 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1236 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1116 ms
ok 116 variable interpolation # in 1644 ms
ok 117 searching solr without credentials fails with error # in 1077 ms
ok 118 --version returns Solr version # in 709 ms
ok 119 version as direct tool call still runs # in 730 ms
ok 120 short help # in 743 ms
ok 121 short help is inferred # in 722 ms
ok 122 long help # in 758 ms
ok 123 running subcommands with zk is prevented # in 698 ms
ok 124 listing out files # in 2099 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 9045 ms
ok 126 copying files around # in 10733 ms
ok 127 upconfig # in 2500 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1163 ms
ok 129 downconfig # in 1141 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1418 ms
ok 131 env var ZK_HOST is honored # in 3112 ms
ok 132 Cleanup # in 1853 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 23s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/xmftflqp6munc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 239 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/239/
No tests ran.
Build Log:
[...truncated 990 lines...]
BUILD FAILED in 29m 18s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/pkotaqd5udltm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 238 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/238/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-19 12:05:02.024; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-19 12:05:02.027; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-19 12:05:02.503; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-19 12:05:02.504; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 16906 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46853 is already being used by another process (pid: 2925526)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-19 12:05:03.668; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-19 12:05:03.672; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 10 post, postlogs and export with basic auth # in 17251 ms
ok 11 package with basic auth # in 12031 ms
ok 12 admin UI static content is excepted from basic auth # in 10401 ms
ok 13 nothing # in 11 ms
ok 14 server does not compress response without Accept-Encoding header # in 175 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 77 ms
ok 16 compressed response can be decompressed and parsed # in 85 ms
ok 17 setting property # in 10848 ms
ok 18 short form of setting property # in 6324 ms
ok 19 connecting to solr via various connection types # in 10613 ms
ok 20 create for non cloud mode # in 7996 ms
ok 21 create for cloud mode # in 9625 ms
ok 22 multiple connection options are prevented # in 860 ms
ok 23 create collection # in 7587 ms
ok 24 create collection using solrUrl # in 5147 ms
ok 25 create collection using legacy solrUrl # in 5177 ms
ok 26 create collection using Zookeeper # in 5354 ms
ok 27 reject d option with invalid config dir # in 2624 ms
ok 28 accept d option with builtin config # in 5184 ms
ok 29 accept d option with explicit path to config # in 5136 ms
ok 30 accept n option as config name # in 5256 ms
ok 31 allow config reuse when n option specifies same config # in 9079 ms
ok 32 create multisharded collections when s provided # in 5195 ms
ok 33 create replicated collections when rf provided # in 6159 ms
ok 34 can delete collections # in 10654 ms
ok 35 can delete collections with solr-url # in 8289 ms
ok 36 can delete collections with solr-connection # in 7888 ms
ok 37 collection delete also deletes zk config # in 7384 ms
ok 38 deletes accompanying zk config with nondefault name # in 7317 ms
ok 39 delete-config option can opt to leave config in zk # in 7109 ms
ok 40 start -e cloud works with --jvm-opts # in 21554 ms
ok 41 start -e cloud works with --no-prompt # in 43307 ms
ok 42 Check export command # in 18900 ms
ok 43 export fails on non cloud mode # in 9208 ms
Waiting for Tika Server to be ready on port 46858
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16588 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16609 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16551 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 10146 ms
ok 48 healthcheck on remote cloud solr # in 9620 ms
ok 49 healthcheck errors on standalone solr # in 11077 ms
ok 50 solr help flag prints help # in 708 ms
ok 51 solr with no flags prints help # in 718 ms
ok 52 start help flag prints help # in 150 ms
ok 53 start h flag prints help # in 141 ms
ok 54 stop help flag prints help # in 139 ms
ok 55 restart help flag prints help # in 144 ms
ok 56 status help flag prints help # in 770 ms
ok 57 healthcheck help flag prints help # in 749 ms
ok 58 create help flag prints help # in 731 ms
ok 59 delete help flag prints help # in 760 ms
ok 60 zk help flag prints help # in 759 ms
ok 61 auth help flag prints help # in 759 ms
ok 62 assert help flag prints help # in 755 ms
ok 63 post help flag prints help # in 745 ms
ok 64 SQL Module # in 15325 ms
ok 65 icu collation in analysis-extras module # in 12254 ms
ok 66 Check lifecycle of sentiment classification # in 22933 ms
ok 67 lifecycle of package # in 8325 ms
ok 68 deploying and undeploying a collection level package # in 13354 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 15083 ms
ok 71 Random placement plugin using ENV # in 15283 ms
ok 72 Check help command # in 2324 ms
ok 73 basic post with a type specified # in 5335 ms
ok 74 basic post with collection instead of url parameter # in 3925 ms
ok 75 basic post with solr-url and collection # in 3790 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 3791 ms
ok 77 basic post WITHOUT a type specified # in 11267 ms
ok 78 crawling a directory as a dry-run # in 797 ms
ok 79 crawling a directory # in 4173 ms
ok 80 skipcommit and optimize and delete # in 3683 ms
ok 81 args mode # in 6884 ms
ok 82 stdin mode # in 3389 ms
ok 83 verbose echo the Solr response # in 3696 ms
ok 84 post solr log into solr via cli # in 9528 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-19T12:15:26.743722495Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-19T12:15:26.743760157Z It should be set to 65000 to avoid operational disruption.
2026-06-19T12:15:26.743763122Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-19T12:15:26.743991452Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-19T12:15:26.743997563Z
2026-06-19T12:15:26.753330474Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-19T12:15:26.761011266Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-19T12:15:26.803817243Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-19T12:15:26.835358723Z WARNING: A command line option has enabled the Security Manager
2026-06-19T12:15:26.835383647Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-19T12:15:27.894007473Z 2026-06-19 12:15:27.890 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-19T12:15:28.155443491Z 2026-06-19 12:15:28.154 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-19T12:15:28.180698181Z 2026-06-19 12:15:28.180 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-19T12:15:28.187590970Z 2026-06-19 12:15:28.187 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-19T12:15:28.187688032Z 2026-06-19 12:15:28.187 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-19T12:15:28.187795637Z 2026-06-19 12:15:28.187 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-19T12:15:28.187940785Z 2026-06-19 12:15:28.187 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-19T12:15:28.187389137Z
2026-06-19T12:15:28.190505277Z 2026-06-19 12:15:28.190 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-19T12:15:28.211608910Z 2026-06-19 12:15:28.211 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-19T12:15:28.224698832Z 2026-06-19 12:15:28.224 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-19T12:15:28.224812999Z 2026-06-19 12:15:28.224 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-19T12:15:28.259203382Z 2026-06-19 12:15:28.258 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-19T12:15:29.147327693Z 2026-06-19 12:15:29.146 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-19T12:15:29.390178742Z 2026-06-19 12:15:29.389 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-19T12:15:29.395460198Z 2026-06-19 12:15:29.395 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-19T12:15:29.395586729Z 2026-06-19 12:15:29.395 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-19T12:15:29.398255373Z 2026-06-19 12:15:29.397 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-19T12:15:29.398288977Z 2026-06-19 12:15:29.397 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-19T12:15:29.441327461Z 2026-06-19 12:15:29.440 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-19T12:15:29.902085449Z 2026-06-19 12:15:29.901 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-19T12:15:29.914628780Z 2026-06-19 12:15:29.914 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-19T12:15:29.955697297Z 2026-06-19 12:15:29.954 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-19T12:15:29.977832625Z 2026-06-19 12:15:29.977 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-19T12:15:29.978070236Z 2026-06-19 12:15:29.977 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-19T12:15:30.109826044Z 2026-06-19 12:15:30.108 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-19T12:15:30.131102855Z 2026-06-19 12:15:30.130 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-19T12:15:30.142111782Z 2026-06-19 12:15:30.141 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-19T12:15:30.153119225Z 2026-06-19 12:15:30.152 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72149538921578496-solr-node1:8983_solr-n_0000000000) starting
2026-06-19T12:15:30.217969461Z 2026-06-19 12:15:30.217 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-19T12:15:30.218362663Z 2026-06-19 12:15:30.218 INFO (OverseerStateUpdate-72149538921578496-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-19T12:15:30.230458819Z 2026-06-19 12:15:30.230 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-19T12:15:30.273200141Z 2026-06-19 12:15:30.272 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-19T12:15:30.691133618Z 2026-06-19 12:15:30.690 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-19T12:15:30.870035056Z 2026-06-19 12:15:30.869 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-19T12:15:31.233311698Z 2026-06-19 12:15:31.232 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-19T12:15:31.364859894Z 2026-06-19 12:15:31.364 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-19T12:15:31.370623029Z 2026-06-19 12:15:31.370 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_19.request.log
2026-06-19T12:15:31.376890213Z 2026-06-19 12:15:31.376 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-19T12:15:31.377822809Z 2026-06-19 12:15:31.377 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4618ms
2026-06-19T12:15:31.555074736Z 2026-06-19 12:15:31.554 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=43
2026-06-19T12:15:31.679981805Z 2026-06-19 12:15:31.679 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-06-19T12:15:35.037593999Z 2026-06-19 12:15:35.037 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-19T12:15:35.155521005Z 2026-06-19 12:15:35.155 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-19T12:15:39.000305530Z 2026-06-19 12:15:38.999 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-06-19T12:15:39.026244856Z 2026-06-19 12:15:39.025 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-06-19T12:15:39.053701020Z 2026-06-19 12:15:39.053 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-06-19T12:15:39.328044972Z 2026-06-19 12:15:39.327 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-19T12:15:39.345202488Z 2026-06-19 12:15:39.344 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-19T12:15:39.352383047Z 2026-06-19 12:15:39.351 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-19T12:15:39.388505104Z 2026-06-19 12:15:39.387 INFO (OverseerStateUpdate-72149538921578496-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-19T12:15:39.388524951Z "core":"test-collection_shard3_replica_n1",
2026-06-19T12:15:39.388528264Z "node_name":"solr-node1:8983_solr",
2026-06-19T12:15:39.388531110Z "base_url":"http://solr-node1:8983/solr",
2026-06-19T12:15:39.388533738Z "collection":"test-collection",
2026-06-19T12:15:39.388536463Z "shard":"shard3",
2026-06-19T12:15:39.388539039Z "state":"down",
2026-06-19T12:15:39.388541858Z "type":"NRT",
2026-06-19T12:15:39.388544342Z "operation":"ADDREPLICA",
2026-06-19T12:15:39.388546904Z "waitForFinalState":"false"}
2026-06-19T12:15:39.394077396Z 2026-06-19 12:15:39.393 INFO (OverseerStateUpdate-72149538921578496-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-19T12:15:39.394097082Z "core":"test-collection_shard2_replica_n2",
2026-06-19T12:15:39.394111677Z "node_name":"solr-node3:8985_solr",
2026-06-19T12:15:39.394115911Z "base_url":"http://solr-node3:8985/solr",
2026-06-19T12:15:39.394119906Z "collection":"test-collection",
2026-06-19T12:15:39.394123546Z "shard":"shard2",
2026-06-19T12:15:39.394126752Z "state":"down",
2026-06-19T12:15:39.394130711Z "type":"NRT",
2026-06-19T12:15:39.394134486Z "operation":"ADDREPLICA",
2026-06-19T12:15:39.394138198Z "waitForFinalState":"false"}
2026-06-19T12:15:39.402234539Z 2026-06-19 12:15:39.401 INFO (OverseerStateUpdate-72149538921578496-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-19T12:15:39.402249727Z "core":"test-collection_shard1_replica_n4",
2026-06-19T12:15:39.402253125Z "node_name":"solr-node2:8984_solr",
2026-06-19T12:15:39.402267721Z "base_url":"http://solr-node2:8984/solr",
2026-06-19T12:15:39.402269930Z "collection":"test-collection",
2026-06-19T12:15:39.402272112Z "shard":"shard1",
2026-06-19T12:15:39.402274017Z "state":"down",
2026-06-19T12:15:39.402275893Z "type":"NRT",
2026-06-19T12:15:39.402277780Z "operation":"ADDREPLICA",
2026-06-19T12:15:39.402279902Z "waitForFinalState":"false"}
2026-06-19T12:15:39.511473160Z 2026-06-19 12:15:39.510 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-19T12:15:39.572735090Z 2026-06-19 12:15:39.572 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-19T12:15:39.612822221Z 2026-06-19 12:15:39.611 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-19T12:15:39.707552747Z 2026-06-19 12:15:39.707 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-19T12:15:40.010868921Z 2026-06-19 12:15:40.010 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-19T12:15:40.010936691Z 2026-06-19 12:15:40.010 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-19T12:15:40.029191950Z 2026-06-19 12:15:40.028 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-06-19T12:15:40.662595468Z 2026-06-19 12:15:40.659 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-19T12:15:40.697381370Z 2026-06-19 12:15:40.696 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-19T12:15:40.705286812Z 2026-06-19 12:15:40.704 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-19T12:15:40.706989167Z 2026-06-19 12:15:40.706 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-19T12:15:40.707282592Z 2026-06-19 12:15:40.706 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-19T12:15:40.764218673Z 2026-06-19 12:15:40.763 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-19T12:15:40.768079436Z 2026-06-19 12:15:40.767 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-19T12:15:40.769843015Z 2026-06-19 12:15:40.769 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-19T12:15:40.772040318Z 2026-06-19 12:15:40.771 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-19T12:15:40.787000227Z 2026-06-19 12:15:40.786 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-19T12:15:40.794846014Z 2026-06-19 12:15:40.794 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-19T12:15:40.802085721Z 2026-06-19 12:15:40.801 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-19T12:15:40.802104355Z 2026-06-19 12:15:40.801 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-19T12:15:40.809307246Z 2026-06-19 12:15:40.808 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-19T12:15:40.814351438Z 2026-06-19 12:15:40.813 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-19T12:15:40.815671718Z 2026-06-19 12:15:40.815 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-19T12:15:40.830508306Z 2026-06-19 12:15:40.830 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-19T12:15:40.830532291Z 2026-06-19 12:15:40.830 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-19T12:15:40.830536290Z 2026-06-19 12:15:40.830 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-06-19T12:15:40.831827926Z 2026-06-19 12:15:40.831 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-19T12:15:40.832333076Z 2026-06-19 12:15:40.832 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-06-19T12:15:40.833864887Z 2026-06-19 12:15:40.833 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72149538921578496-core_node3-n_0000000000
2026-06-19T12:15:40.840331768Z 2026-06-19 12:15:40.840 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-06-19T12:15:40.945106453Z 2026-06-19 12:15:40.944 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 235] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-19T12:15:40.949847767Z 2026-06-19 12:15:40.949 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-19T12:15:40.958258435Z 2026-06-19 12:15:40.957 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1386
2026-06-19T12:15:41.065441636Z 2026-06-19 12:15:41.064 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2956765,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-19T12:15:26.447953346Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-19T12:15:32.016002528Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-19T12:15:32.016031307Z It should be set to 65000 to avoid operational disruption.
2026-06-19T12:15:32.016034301Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-19T12:15:32.016320505Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-19T12:15:32.016333427Z
2026-06-19T12:15:32.025659628Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-19T12:15:32.033182173Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-19T12:15:32.077887142Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-19T12:15:32.115591567Z WARNING: A command line option has enabled the Security Manager
2026-06-19T12:15:32.115609664Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-19T12:15:33.341795994Z 2026-06-19 12:15:33.338 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-19T12:15:33.613610941Z 2026-06-19 12:15:33.613 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-19T12:15:33.640091810Z 2026-06-19 12:15:33.639 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-19T12:15:33.647896224Z 2026-06-19 12:15:33.647 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-19T12:15:33.647919191Z 2026-06-19 12:15:33.647 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-19T12:15:33.648006067Z 2026-06-19 12:15:33.647 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-19T12:15:33.648135699Z 2026-06-19 12:15:33.647 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-19T12:15:33.647539544Z
2026-06-19T12:15:33.651091103Z 2026-06-19 12:15:33.650 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-19T12:15:33.674830068Z 2026-06-19 12:15:33.674 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-19T12:15:33.702118045Z 2026-06-19 12:15:33.701 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-19T12:15:33.848100555Z 2026-06-19 12:15:33.847 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-19T12:15:33.867839602Z 2026-06-19 12:15:33.867 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-19T12:15:33.867955903Z 2026-06-19 12:15:33.867 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-19T12:15:33.868199594Z 2026-06-19 12:15:33.867 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-19T12:15:33.978198713Z 2026-06-19 12:15:33.977 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-19T12:15:33.978471393Z 2026-06-19 12:15:33.978 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-19T12:15:33.978573181Z 2026-06-19 12:15:33.978 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-19T12:15:34.012631152Z 2026-06-19 12:15:34.012 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-19T12:15:34.874338191Z 2026-06-19 12:15:34.873 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-19T12:15:35.013229214Z 2026-06-19 12:15:35.012 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-19T12:15:35.014730409Z 2026-06-19 12:15:35.012 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-19T12:15:35.026518249Z 2026-06-19 12:15:35.026 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-19T12:15:35.032751167Z 2026-06-19 12:15:35.032 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-19T12:15:35.055532904Z 2026-06-19 12:15:35.055 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-19T12:15:35.055761551Z 2026-06-19 12:15:35.055 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-19T12:15:35.094845062Z 2026-06-19 12:15:35.094 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-19T12:15:35.106439251Z 2026-06-19 12:15:35.105 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-19T12:15:35.126443035Z 2026-06-19 12:15:35.126 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-19T12:15:35.137820822Z 2026-06-19 12:15:35.137 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-19T12:15:35.148143698Z 2026-06-19 12:15:35.147 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-19T12:15:35.156564374Z 2026-06-19 12:15:35.156 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-19T12:15:35.204244228Z 2026-06-19 12:15:35.203 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-19T12:15:35.542245188Z 2026-06-19 12:15:35.541 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-19T12:15:35.718078019Z 2026-06-19 12:15:35.717 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-19T12:15:35.986305748Z 2026-06-19 12:15:35.985 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-19T12:15:36.118515293Z 2026-06-19 12:15:36.117 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-19T12:15:36.123759209Z 2026-06-19 12:15:36.123 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_19.request.log
2026-06-19T12:15:36.128334865Z 2026-06-19 12:15:36.127 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-19T12:15:36.129585437Z 2026-06-19 12:15:36.129 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4097ms
2026-06-19T12:15:36.282290245Z 2026-06-19 12:15:36.281 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=45
2026-06-19T12:15:36.346796572Z 2026-06-19 12:15:36.346 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-06-19T12:15:39.300251969Z 2026-06-19 12:15:39.299 INFO (qtp985324122-58-solr-node2-3) [c: s: r: x: t:solr-node2-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=4
2026-06-19T12:15:39.574439392Z 2026-06-19 12:15:39.573 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-19T12:15:39.621633648Z 2026-06-19 12:15:39.621 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-19T12:15:39.679900712Z 2026-06-19 12:15:39.679 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-19T12:15:39.982588970Z 2026-06-19 12:15:39.982 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-19T12:15:39.982623008Z 2026-06-19 12:15:39.982 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-19T12:15:40.001839469Z 2026-06-19 12:15:40.001 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-19T12:15:40.433829615Z 2026-06-19 12:15:40.433 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-19T12:15:40.453579178Z 2026-06-19 12:15:40.453 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-19T12:15:40.457198776Z 2026-06-19 12:15:40.456 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-19T12:15:40.458248859Z 2026-06-19 12:15:40.457 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-19T12:15:40.458345520Z 2026-06-19 12:15:40.458 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-19T12:15:40.502412376Z 2026-06-19 12:15:40.501 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-19T12:15:40.505608029Z 2026-06-19 12:15:40.505 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-19T12:15:40.507530209Z 2026-06-19 12:15:40.507 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-19T12:15:40.509257072Z 2026-06-19 12:15:40.509 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-19T12:15:40.521025085Z 2026-06-19 12:15:40.520 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-19T12:15:40.528980325Z 2026-06-19 12:15:40.528 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-19T12:15:40.534285207Z 2026-06-19 12:15:40.533 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-19T12:15:40.534357586Z 2026-06-19 12:15:40.534 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-19T12:15:40.540710870Z 2026-06-19 12:15:40.540 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-19T12:15:40.552203611Z 2026-06-19 12:15:40.551 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-19T12:15:40.554055061Z 2026-06-19 12:15:40.553 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-19T12:15:40.567476782Z 2026-06-19 12:15:40.566 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-19T12:15:40.567532433Z 2026-06-19 12:15:40.566 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-19T12:15:40.567626115Z 2026-06-19 12:15:40.567 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-19T12:15:40.568573336Z 2026-06-19 12:15:40.568 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-19T12:15:40.569586507Z 2026-06-19 12:15:40.569 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-19T12:15:40.570626155Z 2026-06-19 12:15:40.570 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72149538921578500-core_node6-n_0000000000
2026-06-19T12:15:40.612895781Z 2026-06-19 12:15:40.612 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-19T12:15:40.723637520Z 2026-06-19 12:15:40.723 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-19T12:15:40.727133132Z 2026-06-19 12:15:40.726 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-19T12:15:40.734864620Z 2026-06-19 12:15:40.734 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1161
2026-06-19T12:15:40.945290131Z 2026-06-19 12:15:40.944 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 235] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-19T12:15:40.974558570Z 2026-06-19 12:15:40.974 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-19T12:15:41.065661327Z 2026-06-19 12:15:41.065 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-19T12:15:41.068904835Z 2026-06-19 12:15:41.068 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=1757
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2957172,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-19T12:15:31.787261034Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-19T12:15:32.215736495Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-19T12:15:32.215779461Z It should be set to 65000 to avoid operational disruption.
2026-06-19T12:15:32.215783282Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-19T12:15:32.216031928Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-19T12:15:32.216047841Z
2026-06-19T12:15:32.225272005Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-19T12:15:32.233332874Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-19T12:15:32.278533085Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-19T12:15:32.310926950Z WARNING: A command line option has enabled the Security Manager
2026-06-19T12:15:32.310950147Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-19T12:15:33.361814010Z 2026-06-19 12:15:33.358 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-19T12:15:33.630235058Z 2026-06-19 12:15:33.629 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-19T12:15:33.655709425Z 2026-06-19 12:15:33.655 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-19T12:15:33.662996316Z 2026-06-19 12:15:33.662 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-19T12:15:33.663055902Z 2026-06-19 12:15:33.662 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-19T12:15:33.663193252Z 2026-06-19 12:15:33.662 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-19T12:15:33.663311742Z 2026-06-19 12:15:33.662 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-19T12:15:33.662770248Z
2026-06-19T12:15:33.666066239Z 2026-06-19 12:15:33.665 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-19T12:15:33.688298073Z 2026-06-19 12:15:33.687 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-19T12:15:33.712342352Z 2026-06-19 12:15:33.711 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-19T12:15:33.852181318Z 2026-06-19 12:15:33.851 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-19T12:15:33.871201846Z 2026-06-19 12:15:33.870 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-19T12:15:33.871426994Z 2026-06-19 12:15:33.870 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-19T12:15:33.871437190Z 2026-06-19 12:15:33.871 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-19T12:15:33.981072804Z 2026-06-19 12:15:33.980 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-19T12:15:33.981341722Z 2026-06-19 12:15:33.981 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-19T12:15:33.981435458Z 2026-06-19 12:15:33.981 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-19T12:15:34.015282491Z 2026-06-19 12:15:34.014 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-19T12:15:34.798074090Z 2026-06-19 12:15:34.797 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-19T12:15:34.930977460Z 2026-06-19 12:15:34.930 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-19T12:15:34.932358282Z 2026-06-19 12:15:34.930 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-19T12:15:34.940889908Z 2026-06-19 12:15:34.940 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-19T12:15:34.945524217Z 2026-06-19 12:15:34.945 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-19T12:15:34.948946418Z 2026-06-19 12:15:34.948 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-19T12:15:34.949026659Z 2026-06-19 12:15:34.948 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-19T12:15:34.978036698Z 2026-06-19 12:15:34.977 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-19T12:15:34.985673495Z 2026-06-19 12:15:34.985 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-19T12:15:35.006078542Z 2026-06-19 12:15:35.005 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-19T12:15:35.017766090Z 2026-06-19 12:15:35.017 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-19T12:15:35.027705974Z 2026-06-19 12:15:35.027 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-19T12:15:35.039012447Z 2026-06-19 12:15:35.038 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-19T12:15:35.088938540Z 2026-06-19 12:15:35.088 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-19T12:15:35.156345965Z 2026-06-19 12:15:35.155 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-19T12:15:35.347497970Z 2026-06-19 12:15:35.346 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-19T12:15:35.516934636Z 2026-06-19 12:15:35.516 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-19T12:15:35.774353712Z 2026-06-19 12:15:35.773 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-19T12:15:35.895229507Z 2026-06-19 12:15:35.894 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-19T12:15:35.900630559Z 2026-06-19 12:15:35.900 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_19.request.log
2026-06-19T12:15:35.903878705Z 2026-06-19 12:15:35.903 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-19T12:15:35.904708822Z 2026-06-19 12:15:35.904 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3672ms
2026-06-19T12:15:37.884196439Z 2026-06-19 12:15:37.877 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=73
2026-06-19T12:15:37.944151746Z 2026-06-19 12:15:37.943 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-06-19T12:15:39.575045342Z 2026-06-19 12:15:39.574 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-19T12:15:39.640517194Z 2026-06-19 12:15:39.640 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-19T12:15:39.719336768Z 2026-06-19 12:15:39.718 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-19T12:15:40.135916469Z 2026-06-19 12:15:40.135 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-19T12:15:40.135948994Z 2026-06-19 12:15:40.135 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-19T12:15:40.154422598Z 2026-06-19 12:15:40.153 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-06-19T12:15:40.614749806Z 2026-06-19 12:15:40.614 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-19T12:15:40.636384526Z 2026-06-19 12:15:40.635 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-19T12:15:40.642864994Z 2026-06-19 12:15:40.642 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-19T12:15:40.643856395Z 2026-06-19 12:15:40.643 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-19T12:15:40.643892404Z 2026-06-19 12:15:40.643 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-19T12:15:40.689866236Z 2026-06-19 12:15:40.689 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-19T12:15:40.692677855Z 2026-06-19 12:15:40.692 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-19T12:15:40.694542655Z 2026-06-19 12:15:40.694 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-19T12:15:40.696480109Z 2026-06-19 12:15:40.696 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-19T12:15:40.709762657Z 2026-06-19 12:15:40.709 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-19T12:15:40.717347867Z 2026-06-19 12:15:40.716 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-19T12:15:40.728651719Z 2026-06-19 12:15:40.728 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-19T12:15:40.728815515Z 2026-06-19 12:15:40.728 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-19T12:15:40.735658164Z 2026-06-19 12:15:40.735 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-19T12:15:40.746292196Z 2026-06-19 12:15:40.745 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-19T12:15:40.747672123Z 2026-06-19 12:15:40.747 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-19T12:15:40.762562549Z 2026-06-19 12:15:40.762 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-19T12:15:40.762581940Z 2026-06-19 12:15:40.762 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-19T12:15:40.762627161Z 2026-06-19 12:15:40.762 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-06-19T12:15:40.763965125Z 2026-06-19 12:15:40.763 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-19T12:15:40.764489646Z 2026-06-19 12:15:40.764 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-06-19T12:15:40.765574972Z 2026-06-19 12:15:40.765 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72149538921578499-core_node5-n_0000000000
2026-06-19T12:15:40.772266025Z 2026-06-19 12:15:40.771 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-06-19T12:15:40.945602429Z 2026-06-19 12:15:40.944 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 235] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-19T12:15:40.950314049Z 2026-06-19 12:15:40.950 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-19T12:15:40.958754848Z 2026-06-19 12:15:40.958 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1385
2026-06-19T12:15:41.066376652Z 2026-06-19 12:15:41.065 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2957345,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-19T12:15:31.997705702Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 19088 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Image is up to date for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 7fa88bd647f97d2a92ec9d966e240e3e15dcb70dd0cafbe8120d81742e64c852
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 5c7bb26e5a73ead60d09ac4b9e622b658c0b81e379c41f647dd69f5d15506f9b
# Starting solr-node2 connected to first node's ZooKeeper
# dbbf48a7db0903fccdd14ef7431cf990c6d90b01040983f2e21c287b81f36909
# Started solr-node3 connected to first node's ZooKeeper
# 3468deb0bdc045016029b0633b79aa1c0eead83051bd3a6ba6bfb036e9307582
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-19 12:15:39.083; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-19 12:15:39.157; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node2:8984/solr] to [http://solr-node2:8984/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 15296 ms
ok 87 snapshot lifecycle # in 4728 ms
ok 88 snapshot list # in 3301 ms
ok 89 snapshot list via --solr-connection # in 3378 ms
ok 90 snapshot describe # in 3268 ms
ok 91 snapshot describe via --solr-connection # in 3273 ms
ok 92 start solr with ssl # in 16019 ms
ok 93 use different hostname when not checking peer-name # in 24653 ms
ok 94 start solr with ssl and auth # in 14590 ms
ok 95 start solr with client truststore and security manager # in 16482 ms
ok 96 start solr with mTLS needed # in 46096 ms
ok 97 start solr with mTLS wanted # in 35267 ms
ok 98 test keystore reload # in 58818 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 15909 ms
ok 100 stop command for single port # in 18140 ms
ok 101 check stop command doesn't hang # in 225 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8289 ms
not ok 103 deprecated system properties converted to modern properties # in 195057 ms
# (from function `assert_file_contains' in file .gradle/node/node_modules/bats-file/src/file.bash, line 566,
# in test file test/test_start_solr.bats, line 86)
# `assert_file_contains "${SOLR_LOGS_DIR}/solr.log" 'You are passing in deprecated system property disable.config.edit and should upgrade to using solr.api.config.edit.enabled instead.'' failed
# Waiting up to 180 seconds to see Solr running on port 46853 [|] [/] [-] [\] [|]
# Started Solr server on port 46853 (pid=2974885). Happy searching!
#
#
# -- file does not contain regex --
# path : /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/solr.log
# --
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-5
ok 104 start with custom jetty options # in 8251 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 9210 ms
not ok 106 bootstrapping a configset # in 9205 ms
# (from function `config_exists' in file test/bats_helper.bash, line 84,
# in test file test/test_start_solr.bats, line 116)
# `config_exists "techproducts"' failed
# Waiting up to 180 seconds to see Solr running on port 46853 [|] [/] [-] [\] [|]
# Started Solr server on port 46853 (pid=2977604). Happy searching!
#
# WARN - 2026-06-19 12:24:19.031; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-19 12:24:19.035; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-8
ok 107 status detects locally running solr # in 9564 ms
ok 108 status with --solr-url from user # in 8034 ms
ok 109 status with --port from user # in 7883 ms
ok 110 multiple connection options are prevented # in 858 ms
ok 111 status with invalid --solr-url from user # in 1291 ms
ok 112 status with --short format # in 195871 ms
ok 113 searching solr via locally executed streaming expression # in 1757 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1254 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1148 ms
ok 116 variable interpolation # in 1676 ms
ok 117 searching solr without credentials fails with error # in 1141 ms
ok 118 --version returns Solr version # in 739 ms
ok 119 version as direct tool call still runs # in 748 ms
ok 120 short help # in 765 ms
ok 121 short help is inferred # in 759 ms
ok 122 long help # in 797 ms
ok 123 running subcommands with zk is prevented # in 748 ms
ok 124 listing out files # in 2138 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 9329 ms
ok 126 copying files around # in 10956 ms
ok 127 upconfig # in 2521 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1212 ms
ok 129 downconfig # in 1168 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1431 ms
ok 131 env var ZK_HOST is honored # in 3278 ms
ok 132 Cleanup # in 1850 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 50s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/4cmkf6tmkei2c
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 237 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/237/
No tests ran.
Build Log:
[...truncated 903 lines...]
BUILD FAILED in 29m 50s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/cknmqe7jfh6vg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 236 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/236/
No tests ran.
Build Log:
[...truncated 984 lines...]
BUILD FAILED in 30m 30s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/zgcxifpkyqhsc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 235 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/235/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-16 11:44:20.150; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-16 11:44:20.154; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-16 11:44:20.612; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-16 11:44:20.612; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1466 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 40331 is already being used by another process (pid: 619304)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-16 11:44:21.943; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-16 11:44:21.948; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 17555 ms
ok 11 package with basic auth # in 11893 ms
ok 12 admin UI static content is excepted from basic auth # in 10858 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 141 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 82 ms
ok 16 compressed response can be decompressed and parsed # in 83 ms
ok 17 setting property # in 11068 ms
ok 18 short form of setting property # in 6295 ms
ok 19 connecting to solr via various connection types # in 10458 ms
ok 20 create for non cloud mode # in 7915 ms
ok 21 create for cloud mode # in 9774 ms
ok 22 multiple connection options are prevented # in 837 ms
ok 23 create collection # in 7789 ms
ok 24 create collection using solrUrl # in 5211 ms
ok 25 create collection using legacy solrUrl # in 5232 ms
ok 26 create collection using Zookeeper # in 5681 ms
ok 27 reject d option with invalid config dir # in 2682 ms
ok 28 accept d option with builtin config # in 5285 ms
ok 29 accept d option with explicit path to config # in 5248 ms
ok 30 accept n option as config name # in 5202 ms
ok 31 allow config reuse when n option specifies same config # in 9197 ms
ok 32 create multisharded collections when s provided # in 5178 ms
ok 33 create replicated collections when rf provided # in 6094 ms
ok 34 can delete collections # in 10519 ms
ok 35 can delete collections with solr-url # in 8082 ms
ok 36 can delete collections with solr-connection # in 7708 ms
ok 37 collection delete also deletes zk config # in 7220 ms
ok 38 deletes accompanying zk config with nondefault name # in 7040 ms
ok 39 delete-config option can opt to leave config in zk # in 6941 ms
ok 40 start -e cloud works with --jvm-opts # in 23379 ms
ok 41 start -e cloud works with --no-prompt # in 45733 ms
ok 42 Check export command # in 18497 ms
ok 43 export fails on non cloud mode # in 9208 ms
Waiting for Tika Server to be ready on port 40336
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16593 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16539 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16568 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 10072 ms
ok 48 healthcheck on remote cloud solr # in 9757 ms
ok 49 healthcheck errors on standalone solr # in 11244 ms
ok 50 solr help flag prints help # in 730 ms
ok 51 solr with no flags prints help # in 739 ms
ok 52 start help flag prints help # in 149 ms
ok 53 start h flag prints help # in 148 ms
ok 54 stop help flag prints help # in 139 ms
ok 55 restart help flag prints help # in 146 ms
ok 56 status help flag prints help # in 756 ms
ok 57 healthcheck help flag prints help # in 739 ms
ok 58 create help flag prints help # in 750 ms
ok 59 delete help flag prints help # in 736 ms
ok 60 zk help flag prints help # in 743 ms
ok 61 auth help flag prints help # in 734 ms
ok 62 assert help flag prints help # in 738 ms
ok 63 post help flag prints help # in 739 ms
ok 64 SQL Module # in 15655 ms
ok 65 icu collation in analysis-extras module # in 12330 ms
ok 66 Check lifecycle of sentiment classification # in 23961 ms
ok 67 lifecycle of package # in 8315 ms
ok 68 deploying and undeploying a collection level package # in 12929 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 15071 ms
ok 71 Random placement plugin using ENV # in 15053 ms
ok 72 Check help command # in 2273 ms
ok 73 basic post with a type specified # in 5216 ms
ok 74 basic post with collection instead of url parameter # in 3911 ms
ok 75 basic post with solr-url and collection # in 3787 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 3904 ms
ok 77 basic post WITHOUT a type specified # in 11135 ms
ok 78 crawling a directory as a dry-run # in 781 ms
ok 79 crawling a directory # in 4531 ms
ok 80 skipcommit and optimize and delete # in 3836 ms
ok 81 args mode # in 6807 ms
ok 82 stdin mode # in 3377 ms
ok 83 verbose echo the Solr response # in 3781 ms
ok 84 post solr log into solr via cli # in 9204 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-16T11:54:40.621378890Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-16T11:54:40.621423473Z It should be set to 65000 to avoid operational disruption.
2026-06-16T11:54:40.621427100Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-16T11:54:40.621675176Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-16T11:54:40.621687786Z
2026-06-16T11:54:40.632058187Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-16T11:54:40.640170218Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-16T11:54:40.687942683Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-16T11:54:40.725610077Z WARNING: A command line option has enabled the Security Manager
2026-06-16T11:54:40.725641013Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-16T11:54:41.963960545Z 2026-06-16 11:54:41.960 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-16T11:54:42.231060521Z 2026-06-16 11:54:42.230 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-16T11:54:42.254953438Z 2026-06-16 11:54:42.254 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-16T11:54:42.261627478Z 2026-06-16 11:54:42.261 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-16T11:54:42.261690244Z 2026-06-16 11:54:42.261 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-16T11:54:42.261856421Z 2026-06-16 11:54:42.261 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-16T11:54:42.261977121Z 2026-06-16 11:54:42.261 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-16T11:54:42.261450481Z
2026-06-16T11:54:42.264460707Z 2026-06-16 11:54:42.264 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-16T11:54:42.284563835Z 2026-06-16 11:54:42.284 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-16T11:54:42.297095480Z 2026-06-16 11:54:42.296 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-16T11:54:42.297191935Z 2026-06-16 11:54:42.296 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-16T11:54:42.329697993Z 2026-06-16 11:54:42.329 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-16T11:54:43.087614904Z 2026-06-16 11:54:43.086 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-16T11:54:43.310633972Z 2026-06-16 11:54:43.310 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-16T11:54:43.315275684Z 2026-06-16 11:54:43.314 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-16T11:54:43.315292826Z 2026-06-16 11:54:43.314 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-16T11:54:43.317480107Z 2026-06-16 11:54:43.317 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-16T11:54:43.317556796Z 2026-06-16 11:54:43.317 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-16T11:54:43.354173711Z 2026-06-16 11:54:43.353 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-16T11:54:43.822320496Z 2026-06-16 11:54:43.820 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-16T11:54:43.835718491Z 2026-06-16 11:54:43.835 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-16T11:54:43.885886438Z 2026-06-16 11:54:43.885 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-16T11:54:43.957103319Z 2026-06-16 11:54:43.956 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-16T11:54:43.957176521Z 2026-06-16 11:54:43.956 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-16T11:54:44.071654636Z 2026-06-16 11:54:44.070 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-16T11:54:44.099706464Z 2026-06-16 11:54:44.099 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-16T11:54:44.113478022Z 2026-06-16 11:54:44.112 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-16T11:54:44.119714862Z 2026-06-16 11:54:44.119 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72132540123054080-solr-node1:8983_solr-n_0000000000) starting
2026-06-16T11:54:44.187415011Z 2026-06-16 11:54:44.186 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-16T11:54:44.189677818Z 2026-06-16 11:54:44.189 INFO (OverseerStateUpdate-72132540123054080-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-16T11:54:44.200563287Z 2026-06-16 11:54:44.200 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-16T11:54:44.242336720Z 2026-06-16 11:54:44.241 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-16T11:54:44.702648423Z 2026-06-16 11:54:44.699 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-16T11:54:44.885136623Z 2026-06-16 11:54:44.884 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-16T11:54:45.163015850Z 2026-06-16 11:54:45.162 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-16T11:54:45.373669370Z 2026-06-16 11:54:45.372 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-16T11:54:45.379715686Z 2026-06-16 11:54:45.379 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_16.request.log
2026-06-16T11:54:45.384465518Z 2026-06-16 11:54:45.384 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-16T11:54:45.386196459Z 2026-06-16 11:54:45.385 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4747ms
2026-06-16T11:54:45.554309747Z 2026-06-16 11:54:45.553 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=45
2026-06-16T11:54:45.607494925Z 2026-06-16 11:54:45.607 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-06-16T11:54:48.895276840Z 2026-06-16 11:54:48.894 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-16T11:54:49.214044784Z 2026-06-16 11:54:49.213 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-16T11:54:52.964752624Z 2026-06-16 11:54:52.963 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-06-16T11:54:52.995908562Z 2026-06-16 11:54:52.995 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-06-16T11:54:53.020852279Z 2026-06-16 11:54:53.020 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=3
2026-06-16T11:54:53.279564325Z 2026-06-16 11:54:53.279 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-16T11:54:53.301591675Z 2026-06-16 11:54:53.300 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-16T11:54:53.310061719Z 2026-06-16 11:54:53.309 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:53.353189096Z 2026-06-16 11:54:53.352 INFO (OverseerStateUpdate-72132540123054080-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-16T11:54:53.353215178Z "core":"test-collection_shard3_replica_n1",
2026-06-16T11:54:53.353218011Z "node_name":"solr-node1:8983_solr",
2026-06-16T11:54:53.353220424Z "base_url":"http://solr-node1:8983/solr",
2026-06-16T11:54:53.353222587Z "collection":"test-collection",
2026-06-16T11:54:53.353224941Z "shard":"shard3",
2026-06-16T11:54:53.353226965Z "state":"down",
2026-06-16T11:54:53.353228943Z "type":"NRT",
2026-06-16T11:54:53.353230807Z "operation":"ADDREPLICA",
2026-06-16T11:54:53.353232730Z "waitForFinalState":"false"}
2026-06-16T11:54:53.360160529Z 2026-06-16 11:54:53.359 INFO (OverseerStateUpdate-72132540123054080-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-16T11:54:53.360180607Z "core":"test-collection_shard2_replica_n2",
2026-06-16T11:54:53.360183360Z "node_name":"solr-node3:8985_solr",
2026-06-16T11:54:53.360185776Z "base_url":"http://solr-node3:8985/solr",
2026-06-16T11:54:53.360187873Z "collection":"test-collection",
2026-06-16T11:54:53.360190402Z "shard":"shard2",
2026-06-16T11:54:53.360192557Z "state":"down",
2026-06-16T11:54:53.360194565Z "type":"NRT",
2026-06-16T11:54:53.360196590Z "operation":"ADDREPLICA",
2026-06-16T11:54:53.360198934Z "waitForFinalState":"false"}
2026-06-16T11:54:53.368832972Z 2026-06-16 11:54:53.368 INFO (OverseerStateUpdate-72132540123054080-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-16T11:54:53.368848689Z "core":"test-collection_shard1_replica_n4",
2026-06-16T11:54:53.368851557Z "node_name":"solr-node2:8984_solr",
2026-06-16T11:54:53.368868316Z "base_url":"http://solr-node2:8984/solr",
2026-06-16T11:54:53.368870926Z "collection":"test-collection",
2026-06-16T11:54:53.368872938Z "shard":"shard1",
2026-06-16T11:54:53.368874847Z "state":"down",
2026-06-16T11:54:53.368876874Z "type":"NRT",
2026-06-16T11:54:53.368878750Z "operation":"ADDREPLICA",
2026-06-16T11:54:53.368880787Z "waitForFinalState":"false"}
2026-06-16T11:54:53.477200332Z 2026-06-16 11:54:53.476 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 187] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:53.550350058Z 2026-06-16 11:54:53.548 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-16T11:54:53.623834267Z 2026-06-16 11:54:53.623 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-16T11:54:53.691981184Z 2026-06-16 11:54:53.691 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-16T11:54:54.001102816Z 2026-06-16 11:54:54.000 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-16T11:54:54.001208446Z 2026-06-16 11:54:54.000 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-16T11:54:54.032937525Z 2026-06-16 11:54:54.032 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-06-16T11:54:54.758032770Z 2026-06-16 11:54:54.756 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-16T11:54:54.785760658Z 2026-06-16 11:54:54.784 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-16T11:54:54.790503967Z 2026-06-16 11:54:54.789 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-16T11:54:54.791662788Z 2026-06-16 11:54:54.791 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-16T11:54:54.791769255Z 2026-06-16 11:54:54.791 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-16T11:54:54.864557723Z 2026-06-16 11:54:54.863 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-16T11:54:54.868673383Z 2026-06-16 11:54:54.868 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-16T11:54:54.870924099Z 2026-06-16 11:54:54.870 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-16T11:54:54.873462803Z 2026-06-16 11:54:54.873 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-16T11:54:54.901194213Z 2026-06-16 11:54:54.900 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-16T11:54:54.910681529Z 2026-06-16 11:54:54.910 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-16T11:54:54.917811550Z 2026-06-16 11:54:54.917 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-16T11:54:54.917833769Z 2026-06-16 11:54:54.917 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-16T11:54:54.928462847Z 2026-06-16 11:54:54.927 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-16T11:54:54.936137292Z 2026-06-16 11:54:54.935 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-16T11:54:54.937602182Z 2026-06-16 11:54:54.937 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-16T11:54:54.950370906Z 2026-06-16 11:54:54.949 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-16T11:54:54.950394131Z 2026-06-16 11:54:54.950 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-16T11:54:54.950456571Z 2026-06-16 11:54:54.950 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-06-16T11:54:54.951838604Z 2026-06-16 11:54:54.951 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-16T11:54:54.952362159Z 2026-06-16 11:54:54.952 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-06-16T11:54:54.953653749Z 2026-06-16 11:54:54.953 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72132540123054080-core_node3-n_0000000000
2026-06-16T11:54:54.960197990Z 2026-06-16 11:54:54.959 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-06-16T11:54:55.066504028Z 2026-06-16 11:54:55.065 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:55.071499264Z 2026-06-16 11:54:55.071 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-16T11:54:55.081361020Z 2026-06-16 11:54:55.080 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1533
2026-06-16T11:54:55.183127649Z 2026-06-16 11:54:55.182 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 653092,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-16T11:54:40.387906515Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-16T11:54:45.941133183Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-16T11:54:45.941172792Z It should be set to 65000 to avoid operational disruption.
2026-06-16T11:54:45.941175915Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-16T11:54:45.941493895Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-16T11:54:45.941504620Z
2026-06-16T11:54:45.951312456Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-16T11:54:45.959397675Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-16T11:54:46.008553498Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-16T11:54:46.046953707Z WARNING: A command line option has enabled the Security Manager
2026-06-16T11:54:46.046976871Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-16T11:54:47.172585183Z 2026-06-16 11:54:47.168 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-16T11:54:47.445390608Z 2026-06-16 11:54:47.444 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-16T11:54:47.472449123Z 2026-06-16 11:54:47.471 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-16T11:54:47.479733874Z 2026-06-16 11:54:47.479 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-16T11:54:47.479823479Z 2026-06-16 11:54:47.479 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-16T11:54:47.479937426Z 2026-06-16 11:54:47.479 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-16T11:54:47.480049684Z 2026-06-16 11:54:47.479 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-16T11:54:47.479426328Z
2026-06-16T11:54:47.482687222Z 2026-06-16 11:54:47.482 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-16T11:54:47.504475741Z 2026-06-16 11:54:47.503 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-16T11:54:47.528831534Z 2026-06-16 11:54:47.528 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-16T11:54:47.674832659Z 2026-06-16 11:54:47.674 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-16T11:54:47.693969272Z 2026-06-16 11:54:47.693 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-16T11:54:47.694030942Z 2026-06-16 11:54:47.693 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-16T11:54:47.694183869Z 2026-06-16 11:54:47.693 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-16T11:54:47.804490660Z 2026-06-16 11:54:47.803 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-16T11:54:47.804632238Z 2026-06-16 11:54:47.804 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-16T11:54:47.804715804Z 2026-06-16 11:54:47.804 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-16T11:54:47.837284894Z 2026-06-16 11:54:47.836 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-16T11:54:48.590794010Z 2026-06-16 11:54:48.590 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-16T11:54:48.770974567Z 2026-06-16 11:54:48.769 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-16T11:54:48.772535618Z 2026-06-16 11:54:48.770 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-16T11:54:48.785080754Z 2026-06-16 11:54:48.783 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-16T11:54:48.791920150Z 2026-06-16 11:54:48.791 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-16T11:54:48.798089784Z 2026-06-16 11:54:48.797 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-16T11:54:48.798175153Z 2026-06-16 11:54:48.797 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-16T11:54:48.832166602Z 2026-06-16 11:54:48.831 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-16T11:54:48.839518521Z 2026-06-16 11:54:48.839 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-16T11:54:48.862048563Z 2026-06-16 11:54:48.861 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-16T11:54:48.874169623Z 2026-06-16 11:54:48.873 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-16T11:54:48.883890535Z 2026-06-16 11:54:48.883 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-16T11:54:48.896610917Z 2026-06-16 11:54:48.896 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-16T11:54:48.977925860Z 2026-06-16 11:54:48.977 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-16T11:54:49.215484018Z 2026-06-16 11:54:49.214 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-16T11:54:49.334548976Z 2026-06-16 11:54:49.333 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-16T11:54:49.507208335Z 2026-06-16 11:54:49.506 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-16T11:54:49.795806385Z 2026-06-16 11:54:49.794 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-16T11:54:49.928294947Z 2026-06-16 11:54:49.927 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-16T11:54:49.933931366Z 2026-06-16 11:54:49.933 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_16.request.log
2026-06-16T11:54:49.938434325Z 2026-06-16 11:54:49.938 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-16T11:54:49.940356927Z 2026-06-16 11:54:49.940 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3982ms
2026-06-16T11:54:50.129086155Z 2026-06-16 11:54:50.128 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=56
2026-06-16T11:54:50.200369996Z 2026-06-16 11:54:50.199 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-06-16T11:54:53.552151933Z 2026-06-16 11:54:53.551 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-16T11:54:53.603788211Z 2026-06-16 11:54:53.603 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-16T11:54:53.702910884Z 2026-06-16 11:54:53.702 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-16T11:54:54.001163529Z 2026-06-16 11:54:54.000 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-16T11:54:54.001198296Z 2026-06-16 11:54:54.000 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-16T11:54:54.022763559Z 2026-06-16 11:54:54.022 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-16T11:54:54.486784407Z 2026-06-16 11:54:54.486 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-16T11:54:54.510134053Z 2026-06-16 11:54:54.509 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-16T11:54:54.514283272Z 2026-06-16 11:54:54.513 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-16T11:54:54.515464586Z 2026-06-16 11:54:54.514 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-16T11:54:54.515746846Z 2026-06-16 11:54:54.515 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-16T11:54:54.562735432Z 2026-06-16 11:54:54.562 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-16T11:54:54.566082598Z 2026-06-16 11:54:54.565 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-16T11:54:54.568255753Z 2026-06-16 11:54:54.567 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-16T11:54:54.571952238Z 2026-06-16 11:54:54.571 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-16T11:54:54.584623899Z 2026-06-16 11:54:54.584 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-16T11:54:54.592910626Z 2026-06-16 11:54:54.592 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-16T11:54:54.600524405Z 2026-06-16 11:54:54.599 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-16T11:54:54.600608963Z 2026-06-16 11:54:54.600 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-16T11:54:54.609020772Z 2026-06-16 11:54:54.608 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-16T11:54:54.624004647Z 2026-06-16 11:54:54.623 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-16T11:54:54.625874563Z 2026-06-16 11:54:54.625 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-16T11:54:54.639154525Z 2026-06-16 11:54:54.638 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-16T11:54:54.639175058Z 2026-06-16 11:54:54.638 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-16T11:54:54.639178364Z 2026-06-16 11:54:54.638 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-16T11:54:54.640950704Z 2026-06-16 11:54:54.640 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-16T11:54:54.641539937Z 2026-06-16 11:54:54.641 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-16T11:54:54.642759049Z 2026-06-16 11:54:54.642 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72132540123054083-core_node6-n_0000000000
2026-06-16T11:54:54.653735360Z 2026-06-16 11:54:54.653 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-16T11:54:54.761293192Z 2026-06-16 11:54:54.760 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 207] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:54.765370685Z 2026-06-16 11:54:54.764 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-16T11:54:54.774460915Z 2026-06-16 11:54:54.773 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1224
2026-06-16T11:54:54.873487991Z 2026-06-16 11:54:54.872 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 221] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:55.066567699Z 2026-06-16 11:54:55.066 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:55.183325377Z 2026-06-16 11:54:55.182 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 653492,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-16T11:54:45.719291079Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-16T11:54:46.162092286Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-16T11:54:46.162149312Z It should be set to 65000 to avoid operational disruption.
2026-06-16T11:54:46.162154838Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-16T11:54:46.162419857Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-16T11:54:46.162431830Z
2026-06-16T11:54:46.172108159Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-16T11:54:46.181096481Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-16T11:54:46.237015110Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-16T11:54:46.273873956Z WARNING: A command line option has enabled the Security Manager
2026-06-16T11:54:46.273909223Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-16T11:54:47.532057262Z 2026-06-16 11:54:47.528 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-16T11:54:47.816308844Z 2026-06-16 11:54:47.815 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-16T11:54:47.842109153Z 2026-06-16 11:54:47.841 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-16T11:54:47.849704484Z 2026-06-16 11:54:47.849 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-16T11:54:47.849829965Z 2026-06-16 11:54:47.849 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-16T11:54:47.850006211Z 2026-06-16 11:54:47.849 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-16T11:54:47.850160990Z 2026-06-16 11:54:47.849 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-16T11:54:47.849502272Z
2026-06-16T11:54:47.852730116Z 2026-06-16 11:54:47.852 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-16T11:54:47.875957658Z 2026-06-16 11:54:47.875 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-16T11:54:47.901859569Z 2026-06-16 11:54:47.901 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-16T11:54:48.041862871Z 2026-06-16 11:54:48.041 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-16T11:54:48.058722741Z 2026-06-16 11:54:48.058 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-16T11:54:48.058791721Z 2026-06-16 11:54:48.058 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-16T11:54:48.058925259Z 2026-06-16 11:54:48.058 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-16T11:54:48.168056749Z 2026-06-16 11:54:48.167 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-16T11:54:48.168144415Z 2026-06-16 11:54:48.167 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-16T11:54:48.168258432Z 2026-06-16 11:54:48.168 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-16T11:54:48.201198348Z 2026-06-16 11:54:48.200 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-16T11:54:48.966380821Z 2026-06-16 11:54:48.965 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-16T11:54:49.106635939Z 2026-06-16 11:54:49.106 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-16T11:54:49.107934861Z 2026-06-16 11:54:49.106 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-16T11:54:49.116325006Z 2026-06-16 11:54:49.115 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-16T11:54:49.121075725Z 2026-06-16 11:54:49.120 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-16T11:54:49.124623647Z 2026-06-16 11:54:49.124 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-16T11:54:49.124702981Z 2026-06-16 11:54:49.124 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-16T11:54:49.158252821Z 2026-06-16 11:54:49.157 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-16T11:54:49.166427447Z 2026-06-16 11:54:49.166 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-16T11:54:49.185467953Z 2026-06-16 11:54:49.184 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-16T11:54:49.196754558Z 2026-06-16 11:54:49.196 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-16T11:54:49.206516085Z 2026-06-16 11:54:49.206 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-16T11:54:49.215003537Z 2026-06-16 11:54:49.214 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-16T11:54:49.262700790Z 2026-06-16 11:54:49.262 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-16T11:54:49.512283805Z 2026-06-16 11:54:49.511 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-16T11:54:49.675340761Z 2026-06-16 11:54:49.674 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-16T11:54:49.927166612Z 2026-06-16 11:54:49.926 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-16T11:54:50.050877497Z 2026-06-16 11:54:50.050 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-16T11:54:50.056193501Z 2026-06-16 11:54:50.055 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_16.request.log
2026-06-16T11:54:50.059815166Z 2026-06-16 11:54:50.059 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-16T11:54:50.060618791Z 2026-06-16 11:54:50.060 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3880ms
2026-06-16T11:54:51.736399536Z 2026-06-16 11:54:51.726 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=79
2026-06-16T11:54:51.803077922Z 2026-06-16 11:54:51.802 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-06-16T11:54:53.245365195Z 2026-06-16 11:54:53.244 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=9
2026-06-16T11:54:53.551496368Z 2026-06-16 11:54:53.550 INFO (qtp985324122-30-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-16T11:54:53.632915930Z 2026-06-16 11:54:53.632 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-16T11:54:53.738538034Z 2026-06-16 11:54:53.737 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-16T11:54:54.124772044Z 2026-06-16 11:54:54.124 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-16T11:54:54.124816158Z 2026-06-16 11:54:54.124 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-16T11:54:54.147836957Z 2026-06-16 11:54:54.147 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-06-16T11:54:54.655631704Z 2026-06-16 11:54:54.655 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-16T11:54:54.733853986Z 2026-06-16 11:54:54.733 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-16T11:54:54.737439100Z 2026-06-16 11:54:54.737 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-16T11:54:54.738620481Z 2026-06-16 11:54:54.737 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-16T11:54:54.738634456Z 2026-06-16 11:54:54.738 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-16T11:54:54.785908337Z 2026-06-16 11:54:54.785 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-16T11:54:54.789387681Z 2026-06-16 11:54:54.789 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-16T11:54:54.791419526Z 2026-06-16 11:54:54.791 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-16T11:54:54.793324829Z 2026-06-16 11:54:54.793 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-16T11:54:54.806595506Z 2026-06-16 11:54:54.806 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-16T11:54:54.815125262Z 2026-06-16 11:54:54.814 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-16T11:54:54.821715020Z 2026-06-16 11:54:54.821 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-16T11:54:54.821804322Z 2026-06-16 11:54:54.821 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-16T11:54:54.828663312Z 2026-06-16 11:54:54.828 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-16T11:54:54.855077119Z 2026-06-16 11:54:54.854 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-16T11:54:54.856747131Z 2026-06-16 11:54:54.856 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-16T11:54:54.870591298Z 2026-06-16 11:54:54.870 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-16T11:54:54.870665436Z 2026-06-16 11:54:54.870 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-16T11:54:54.870670233Z 2026-06-16 11:54:54.870 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-06-16T11:54:54.874232012Z 2026-06-16 11:54:54.873 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-16T11:54:54.874883419Z 2026-06-16 11:54:54.874 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-06-16T11:54:54.876323948Z 2026-06-16 11:54:54.876 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72132540123054084-core_node5-n_0000000000
2026-06-16T11:54:54.886181253Z 2026-06-16 11:54:54.885 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-06-16T11:54:55.066713650Z 2026-06-16 11:54:55.066 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:55.072029312Z 2026-06-16 11:54:55.071 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-16T11:54:55.083015696Z 2026-06-16 11:54:55.082 INFO (qtp985324122-30-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1533
2026-06-16T11:54:55.094532324Z 2026-06-16 11:54:55.094 INFO (qtp985324122-27-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-16T11:54:55.183745799Z 2026-06-16 11:54:55.183 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-16T11:54:55.187518607Z 2026-06-16 11:54:55.187 INFO (qtp985324122-27-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=1928
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 653674,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-16T11:54:45.92901938Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 25644 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 66b76b382631: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Download complete
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 3aa9120e46f5: Verifying Checksum
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Already exists
# 9d5a0b6cb36c: Already exists
# 49518c28c0b3: Already exists
# 41162acdcc29: Already exists
# 2ed0fd558ca3: Already exists
# 2488415fda1c: Pulling fs layer
# 09511c8fb203: Pulling fs layer
# a0c57fb89c4f: Pulling fs layer
# d0641dcc41d7: Pulling fs layer
# 315722fd9570: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# d0641dcc41d7: Waiting
# 315722fd9570: Waiting
# 4f4fb700ef54: Waiting
# 09511c8fb203: Verifying Checksum
# 09511c8fb203: Download complete
# a0c57fb89c4f: Download complete
# d0641dcc41d7: Verifying Checksum
# d0641dcc41d7: Download complete
# 315722fd9570: Verifying Checksum
# 315722fd9570: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 2488415fda1c: Verifying Checksum
# 2488415fda1c: Download complete
# 2488415fda1c: Pull complete
# 09511c8fb203: Pull complete
# a0c57fb89c4f: Pull complete
# d0641dcc41d7: Pull complete
# 315722fd9570: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 3bb5dc8347b91ee7b001cd47c2807e4b5a7fc9d99e5251e4d1f5a8f9c9dbcc06
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 02ec503a477a0f1d51aafee998f314445eb9e2315d5a217a69fd194476f60678
# Starting solr-node2 connected to first node's ZooKeeper
# b8cb7d11a901b37efc1b8b9f69e5e90aa4a53e46d3725648a72af837b6b75c55
# Started solr-node3 connected to first node's ZooKeeper
# bc2b0c27ef6004a5b0c1303aec16afc0bd7d56615c570f4f6767ffa10d0040b1
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-16 11:54:53.063; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-16 11:54:53.097; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 16504 ms
ok 87 snapshot lifecycle # in 5234 ms
ok 88 snapshot list # in 3653 ms
ok 89 snapshot list via --solr-connection # in 3502 ms
ok 90 snapshot describe # in 3343 ms
ok 91 snapshot describe via --solr-connection # in 3304 ms
ok 92 start solr with ssl # in 16377 ms
ok 93 use different hostname when not checking peer-name # in 24782 ms
ok 94 start solr with ssl and auth # in 14681 ms
ok 95 start solr with client truststore and security manager # in 16432 ms
ok 96 start solr with mTLS needed # in 46414 ms
ok 97 start solr with mTLS wanted # in 34299 ms
ok 98 test keystore reload # in 58602 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 15987 ms
ok 100 stop command for single port # in 18004 ms
ok 101 check stop command doesn't hang # in 219 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8216 ms
not ok 103 deprecated system properties converted to modern properties # in 195225 ms
# (from function `assert_file_contains' in file .gradle/node/node_modules/bats-file/src/file.bash, line 566,
# in test file test/test_start_solr.bats, line 86)
# `assert_file_contains "${SOLR_LOGS_DIR}/solr.log" 'You are passing in deprecated system property disable.config.edit and should upgrade to using solr.api.config.edit.enabled instead.'' failed
# Waiting up to 180 seconds to see Solr running on port 40331 [|] [/] [-] [\] [|]
# Started Solr server on port 40331 (pid=671180). Happy searching!
#
#
# -- file does not contain regex --
# path : /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/solr.log
# --
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-5
ok 104 start with custom jetty options # in 8166 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 9193 ms
not ok 106 bootstrapping a configset # in 9259 ms
# (from function `config_exists' in file test/bats_helper.bash, line 84,
# in test file test/test_start_solr.bats, line 116)
# `config_exists "techproducts"' failed
# Waiting up to 180 seconds to see Solr running on port 40331 [|] [/] [-] [\] [|]
# Started Solr server on port 40331 (pid=673907). Happy searching!
#
# WARN - 2026-06-16 12:03:35.557; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-16 12:03:35.562; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-8
ok 107 status detects locally running solr # in 9626 ms
ok 108 status with --solr-url from user # in 7934 ms
ok 109 status with --port from user # in 7924 ms
ok 110 multiple connection options are prevented # in 843 ms
ok 111 status with invalid --solr-url from user # in 1301 ms
ok 112 status with --short format # in 195935 ms
ok 113 searching solr via locally executed streaming expression # in 1737 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1276 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1108 ms
ok 116 variable interpolation # in 1717 ms
ok 117 searching solr without credentials fails with error # in 1134 ms
ok 118 --version returns Solr version # in 748 ms
ok 119 version as direct tool call still runs # in 748 ms
ok 120 short help # in 803 ms
ok 121 short help is inferred # in 741 ms
ok 122 long help # in 763 ms
ok 123 running subcommands with zk is prevented # in 716 ms
ok 124 listing out files # in 2112 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 9356 ms
ok 126 copying files around # in 10804 ms
ok 127 upconfig # in 2641 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1089 ms
ok 129 downconfig # in 1143 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1399 ms
ok 131 env var ZK_HOST is honored # in 3191 ms
ok 132 Cleanup # in 1938 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 46s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/k35tb4obip2zw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 234 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/234/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-15 12:09:00.069; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-15 12:09:00.074; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-15 12:09:00.628; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-15 12:09:00.628; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1531 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38375 is already being used by another process (pid: 3853610)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-15 12:09:02.010; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-15 12:09:02.015; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
not ok 10 post, postlogs and export with basic auth # in 16772 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38375 is already being used by another process (pid: 3853610)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/10-2
# Last output:
# WARN - 2026-06-15 12:09:03.424; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-15 12:09:03.428; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 11 package with basic auth # in 12940 ms
ok 12 admin UI static content is excepted from basic auth # in 11110 ms
ok 13 nothing # in 11 ms
ok 14 server does not compress response without Accept-Encoding header # in 162 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 87 ms
ok 16 compressed response can be decompressed and parsed # in 88 ms
ok 17 setting property # in 12222 ms
ok 18 short form of setting property # in 6812 ms
ok 19 connecting to solr via various connection types # in 11554 ms
ok 20 create for non cloud mode # in 8406 ms
ok 21 create for cloud mode # in 10595 ms
ok 22 multiple connection options are prevented # in 929 ms
ok 23 create collection # in 8438 ms
ok 24 create collection using solrUrl # in 5542 ms
ok 25 create collection using legacy solrUrl # in 5432 ms
ok 26 create collection using Zookeeper # in 5603 ms
ok 27 reject d option with invalid config dir # in 2810 ms
ok 28 accept d option with builtin config # in 5518 ms
ok 29 accept d option with explicit path to config # in 5550 ms
ok 30 accept n option as config name # in 5441 ms
ok 31 allow config reuse when n option specifies same config # in 9415 ms
ok 32 create multisharded collections when s provided # in 5639 ms
ok 33 create replicated collections when rf provided # in 6379 ms
ok 34 can delete collections # in 11638 ms
ok 35 can delete collections with solr-url # in 8837 ms
ok 36 can delete collections with solr-connection # in 8403 ms
ok 37 collection delete also deletes zk config # in 7774 ms
ok 38 deletes accompanying zk config with nondefault name # in 7879 ms
ok 39 delete-config option can opt to leave config in zk # in 7904 ms
ok 40 start -e cloud works with --jvm-opts # in 22881 ms
ok 41 start -e cloud works with --no-prompt # in 45563 ms
ok 42 Check export command # in 20315 ms
ok 43 export fails on non cloud mode # in 9943 ms
Waiting for Tika Server to be ready on port 38380
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16674 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16631 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16632 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 10874 ms
ok 48 healthcheck on remote cloud solr # in 10668 ms
ok 49 healthcheck errors on standalone solr # in 11962 ms
ok 50 solr help flag prints help # in 813 ms
ok 51 solr with no flags prints help # in 806 ms
ok 52 start help flag prints help # in 165 ms
ok 53 start h flag prints help # in 155 ms
ok 54 stop help flag prints help # in 148 ms
ok 55 restart help flag prints help # in 164 ms
ok 56 status help flag prints help # in 837 ms
ok 57 healthcheck help flag prints help # in 810 ms
ok 58 create help flag prints help # in 822 ms
ok 59 delete help flag prints help # in 824 ms
ok 60 zk help flag prints help # in 804 ms
ok 61 auth help flag prints help # in 811 ms
ok 62 assert help flag prints help # in 843 ms
ok 63 post help flag prints help # in 822 ms
ok 64 SQL Module # in 16710 ms
ok 65 icu collation in analysis-extras module # in 13018 ms
ok 66 Check lifecycle of sentiment classification # in 24791 ms
ok 67 lifecycle of package # in 8662 ms
ok 68 deploying and undeploying a collection level package # in 13931 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 16857 ms
ok 71 Random placement plugin using ENV # in 16647 ms
ok 72 Check help command # in 2579 ms
ok 73 basic post with a type specified # in 5671 ms
ok 74 basic post with collection instead of url parameter # in 4241 ms
ok 75 basic post with solr-url and collection # in 4193 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 4279 ms
ok 77 basic post WITHOUT a type specified # in 12612 ms
ok 78 crawling a directory as a dry-run # in 884 ms
ok 79 crawling a directory # in 4653 ms
ok 80 skipcommit and optimize and delete # in 4185 ms
ok 81 args mode # in 7480 ms
ok 82 stdin mode # in 3688 ms
ok 83 verbose echo the Solr response # in 4038 ms
ok 84 post solr log into solr via cli # in 10597 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-15T12:19:56.279323243Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-15T12:19:56.279377896Z It should be set to 65000 to avoid operational disruption.
2026-06-15T12:19:56.279381714Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-15T12:19:56.279700350Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-15T12:19:56.279712693Z
2026-06-15T12:19:56.289390493Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-15T12:19:56.297462986Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-15T12:19:56.344562244Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-15T12:19:56.378271454Z WARNING: A command line option has enabled the Security Manager
2026-06-15T12:19:56.378359210Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-15T12:19:57.637988777Z 2026-06-15 12:19:57.634 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-15T12:19:57.952791352Z 2026-06-15 12:19:57.952 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-15T12:19:57.978318851Z 2026-06-15 12:19:57.977 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-15T12:19:57.985425905Z 2026-06-15 12:19:57.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-15T12:19:57.985507968Z 2026-06-15 12:19:57.985 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-15T12:19:57.985588619Z 2026-06-15 12:19:57.985 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-15T12:19:57.985705495Z 2026-06-15 12:19:57.985 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-15T12:19:57.985206321Z
2026-06-15T12:19:57.988154693Z 2026-06-15 12:19:57.987 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-15T12:19:58.010625766Z 2026-06-15 12:19:58.010 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-15T12:19:58.025094285Z 2026-06-15 12:19:58.024 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-15T12:19:58.025166162Z 2026-06-15 12:19:58.024 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-15T12:19:58.062712112Z 2026-06-15 12:19:58.062 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-15T12:19:58.803348248Z 2026-06-15 12:19:58.802 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-15T12:19:59.050904403Z 2026-06-15 12:19:59.050 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-15T12:19:59.056168458Z 2026-06-15 12:19:59.055 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-15T12:19:59.056288311Z 2026-06-15 12:19:59.055 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-15T12:19:59.058825592Z 2026-06-15 12:19:59.058 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-15T12:19:59.058946848Z 2026-06-15 12:19:59.058 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-15T12:19:59.096642185Z 2026-06-15 12:19:59.096 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-15T12:19:59.567723528Z 2026-06-15 12:19:59.565 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-15T12:19:59.588025592Z 2026-06-15 12:19:59.587 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-15T12:19:59.631673474Z 2026-06-15 12:19:59.631 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-15T12:19:59.746847653Z 2026-06-15 12:19:59.679 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-15T12:19:59.746868347Z 2026-06-15 12:19:59.746 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-15T12:19:59.918486387Z 2026-06-15 12:19:59.917 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-15T12:19:59.943072405Z 2026-06-15 12:19:59.942 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-15T12:19:59.960330624Z 2026-06-15 12:19:59.959 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-15T12:19:59.967133991Z 2026-06-15 12:19:59.966 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72126907354775552-solr-node1:8983_solr-n_0000000000) starting
2026-06-15T12:20:00.038552407Z 2026-06-15 12:20:00.037 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-15T12:20:00.038667570Z 2026-06-15 12:20:00.037 INFO (OverseerStateUpdate-72126907354775552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-15T12:20:00.053049675Z 2026-06-15 12:20:00.051 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-15T12:20:00.104187028Z 2026-06-15 12:20:00.103 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-15T12:20:00.592869314Z 2026-06-15 12:20:00.590 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-15T12:20:00.775069364Z 2026-06-15 12:20:00.774 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-15T12:20:01.054238273Z 2026-06-15 12:20:01.053 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-15T12:20:01.191613250Z 2026-06-15 12:20:01.190 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-15T12:20:01.198523730Z 2026-06-15 12:20:01.198 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_15.request.log
2026-06-15T12:20:01.293758576Z 2026-06-15 12:20:01.292 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-15T12:20:01.294760216Z 2026-06-15 12:20:01.294 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4998ms
2026-06-15T12:20:01.496114573Z 2026-06-15 12:20:01.494 INFO (qtp985324122-28-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=50
2026-06-15T12:20:01.563465522Z 2026-06-15 12:20:01.562 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-06-15T12:20:05.226343833Z 2026-06-15 12:20:05.225 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-15T12:20:05.654088393Z 2026-06-15 12:20:05.653 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-15T12:20:10.179311543Z 2026-06-15 12:20:10.178 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-06-15T12:20:10.215159544Z 2026-06-15 12:20:10.214 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-06-15T12:20:10.250394337Z 2026-06-15 12:20:10.249 INFO (qtp985324122-28-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-06-15T12:20:10.582419938Z 2026-06-15 12:20:10.581 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-15T12:20:10.602728590Z 2026-06-15 12:20:10.602 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-15T12:20:10.609544273Z 2026-06-15 12:20:10.609 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:10.651728356Z 2026-06-15 12:20:10.651 INFO (OverseerStateUpdate-72126907354775552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-15T12:20:10.651751390Z "core":"test-collection_shard2_replica_n1",
2026-06-15T12:20:10.651754904Z "node_name":"solr-node3:8985_solr",
2026-06-15T12:20:10.651757663Z "base_url":"http://solr-node3:8985/solr",
2026-06-15T12:20:10.651760365Z "collection":"test-collection",
2026-06-15T12:20:10.651763026Z "shard":"shard2",
2026-06-15T12:20:10.651765710Z "state":"down",
2026-06-15T12:20:10.651768237Z "type":"NRT",
2026-06-15T12:20:10.651770750Z "operation":"ADDREPLICA",
2026-06-15T12:20:10.651773358Z "waitForFinalState":"false"}
2026-06-15T12:20:10.661054397Z 2026-06-15 12:20:10.660 INFO (OverseerStateUpdate-72126907354775552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-15T12:20:10.661074023Z "core":"test-collection_shard1_replica_n2",
2026-06-15T12:20:10.661077119Z "node_name":"solr-node2:8984_solr",
2026-06-15T12:20:10.661079788Z "base_url":"http://solr-node2:8984/solr",
2026-06-15T12:20:10.661082466Z "collection":"test-collection",
2026-06-15T12:20:10.661085099Z "shard":"shard1",
2026-06-15T12:20:10.661087758Z "state":"down",
2026-06-15T12:20:10.661090362Z "type":"NRT",
2026-06-15T12:20:10.661092962Z "operation":"ADDREPLICA",
2026-06-15T12:20:10.661106645Z "waitForFinalState":"false"}
2026-06-15T12:20:10.668169237Z 2026-06-15 12:20:10.667 INFO (OverseerStateUpdate-72126907354775552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-15T12:20:10.668186173Z "core":"test-collection_shard3_replica_n4",
2026-06-15T12:20:10.668189138Z "node_name":"solr-node1:8983_solr",
2026-06-15T12:20:10.668207941Z "base_url":"http://solr-node1:8983/solr",
2026-06-15T12:20:10.668210025Z "collection":"test-collection",
2026-06-15T12:20:10.668212002Z "shard":"shard3",
2026-06-15T12:20:10.668213988Z "state":"down",
2026-06-15T12:20:10.668215908Z "type":"NRT",
2026-06-15T12:20:10.668217908Z "operation":"ADDREPLICA",
2026-06-15T12:20:10.668219839Z "waitForFinalState":"false"}
2026-06-15T12:20:10.779533124Z 2026-06-15 12:20:10.778 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:10.888149643Z 2026-06-15 12:20:10.887 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-15T12:20:10.934678248Z 2026-06-15 12:20:10.934 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-15T12:20:11.014641782Z 2026-06-15 12:20:11.014 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-15T12:20:11.390222008Z 2026-06-15 12:20:11.379 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-15T12:20:11.390245206Z 2026-06-15 12:20:11.388 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-15T12:20:11.422137314Z 2026-06-15 12:20:11.421 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n4], dataDir=[/var/solr/data/test-collection_shard3_replica_n4/data/]
2026-06-15T12:20:12.316381395Z 2026-06-15 12:20:12.314 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-15T12:20:12.347273118Z 2026-06-15 12:20:12.346 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-15T12:20:12.353541688Z 2026-06-15 12:20:12.352 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-15T12:20:12.354112581Z 2026-06-15 12:20:12.353 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-15T12:20:12.354210072Z 2026-06-15 12:20:12.353 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-15T12:20:12.423323755Z 2026-06-15 12:20:12.422 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-15T12:20:12.426397680Z 2026-06-15 12:20:12.426 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-15T12:20:12.429141592Z 2026-06-15 12:20:12.428 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-15T12:20:12.431154990Z 2026-06-15 12:20:12.430 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-15T12:20:12.468290765Z 2026-06-15 12:20:12.467 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-15T12:20:12.478449173Z 2026-06-15 12:20:12.477 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-15T12:20:12.489109203Z 2026-06-15 12:20:12.488 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-15T12:20:12.489143640Z 2026-06-15 12:20:12.488 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-15T12:20:12.497571040Z 2026-06-15 12:20:12.496 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-15T12:20:12.509202335Z 2026-06-15 12:20:12.508 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-15T12:20:12.511125575Z 2026-06-15 12:20:12.510 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-15T12:20:12.527444421Z 2026-06-15 12:20:12.526 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-15T12:20:12.527598852Z 2026-06-15 12:20:12.527 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-15T12:20:12.527675544Z 2026-06-15 12:20:12.527 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n4/
2026-06-15T12:20:12.528977536Z 2026-06-15 12:20:12.528 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-15T12:20:12.529530560Z 2026-06-15 12:20:12.529 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ has no replicas
2026-06-15T12:20:12.530864727Z 2026-06-15 12:20:12.530 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72126907354775552-core_node6-n_0000000000
2026-06-15T12:20:12.539283154Z 2026-06-15 12:20:12.538 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ shard3
2026-06-15T12:20:12.722034846Z 2026-06-15 12:20:12.721 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:12.726038325Z 2026-06-15 12:20:12.725 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-15T12:20:12.743204777Z 2026-06-15 12:20:12.742 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1857
2026-06-15T12:20:12.844902040Z 2026-06-15 12:20:12.844 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3884471,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-15T12:19:56.001290926Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-15T12:20:01.912588605Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-15T12:20:01.912614861Z It should be set to 65000 to avoid operational disruption.
2026-06-15T12:20:01.912617778Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-15T12:20:01.913065904Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-15T12:20:01.913083195Z
2026-06-15T12:20:01.925689892Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-15T12:20:01.935818083Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-15T12:20:01.982041515Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-15T12:20:02.019215860Z WARNING: A command line option has enabled the Security Manager
2026-06-15T12:20:02.019240049Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-15T12:20:03.268103813Z 2026-06-15 12:20:03.264 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-15T12:20:03.549871390Z 2026-06-15 12:20:03.549 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-15T12:20:03.577794759Z 2026-06-15 12:20:03.577 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-15T12:20:03.585737679Z 2026-06-15 12:20:03.584 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-15T12:20:03.585764086Z 2026-06-15 12:20:03.585 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-15T12:20:03.586156018Z 2026-06-15 12:20:03.585 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-15T12:20:03.586173666Z 2026-06-15 12:20:03.585 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-15T12:20:03.585281498Z
2026-06-15T12:20:03.588918030Z 2026-06-15 12:20:03.588 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-15T12:20:03.612551971Z 2026-06-15 12:20:03.611 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-15T12:20:03.637813725Z 2026-06-15 12:20:03.637 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-15T12:20:03.784320369Z 2026-06-15 12:20:03.783 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-15T12:20:03.806457866Z 2026-06-15 12:20:03.805 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-15T12:20:03.806599606Z 2026-06-15 12:20:03.806 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-15T12:20:03.806724885Z 2026-06-15 12:20:03.806 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-15T12:20:03.918495784Z 2026-06-15 12:20:03.917 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-15T12:20:03.918536333Z 2026-06-15 12:20:03.918 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-15T12:20:03.918612760Z 2026-06-15 12:20:03.918 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-15T12:20:03.953040882Z 2026-06-15 12:20:03.952 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-15T12:20:04.790955192Z 2026-06-15 12:20:04.790 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-15T12:20:04.961975755Z 2026-06-15 12:20:04.961 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-15T12:20:04.963371839Z 2026-06-15 12:20:04.961 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-15T12:20:04.976975198Z 2026-06-15 12:20:04.975 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-15T12:20:04.992851673Z 2026-06-15 12:20:04.992 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-15T12:20:04.998839177Z 2026-06-15 12:20:04.998 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-15T12:20:04.998952011Z 2026-06-15 12:20:04.998 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-15T12:20:05.144347133Z 2026-06-15 12:20:05.142 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-15T12:20:05.157456308Z 2026-06-15 12:20:05.156 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-15T12:20:05.182789163Z 2026-06-15 12:20:05.182 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-15T12:20:05.197445506Z 2026-06-15 12:20:05.196 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-15T12:20:05.211293016Z 2026-06-15 12:20:05.210 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-15T12:20:05.228043133Z 2026-06-15 12:20:05.227 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-15T12:20:05.285399139Z 2026-06-15 12:20:05.284 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-15T12:20:05.593401239Z 2026-06-15 12:20:05.590 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-15T12:20:05.654000828Z 2026-06-15 12:20:05.653 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-15T12:20:05.777243480Z 2026-06-15 12:20:05.776 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-15T12:20:06.082083582Z 2026-06-15 12:20:06.081 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-15T12:20:06.218038138Z 2026-06-15 12:20:06.217 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-15T12:20:06.223843484Z 2026-06-15 12:20:06.223 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_15.request.log
2026-06-15T12:20:06.227559443Z 2026-06-15 12:20:06.226 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-15T12:20:06.230756890Z 2026-06-15 12:20:06.230 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4296ms
2026-06-15T12:20:06.414607772Z 2026-06-15 12:20:06.413 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=54
2026-06-15T12:20:06.486908097Z 2026-06-15 12:20:06.485 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-06-15T12:20:10.549230537Z 2026-06-15 12:20:10.548 INFO (qtp985324122-58-solr-node2-3) [c: s: r: x: t:solr-node2-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=6
2026-06-15T12:20:10.886079392Z 2026-06-15 12:20:10.885 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard1_replica_n2&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-15T12:20:10.962926015Z 2026-06-15 12:20:10.962 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-15T12:20:11.166119833Z 2026-06-15 12:20:11.165 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-15T12:20:11.442154778Z 2026-06-15 12:20:11.441 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-15T12:20:11.442308852Z 2026-06-15 12:20:11.441 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-15T12:20:11.465049330Z 2026-06-15 12:20:11.464 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n2], dataDir=[/var/solr/data/test-collection_shard1_replica_n2/data/]
2026-06-15T12:20:11.949871105Z 2026-06-15 12:20:11.948 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-15T12:20:11.973121152Z 2026-06-15 12:20:11.972 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-15T12:20:11.977426746Z 2026-06-15 12:20:11.976 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-15T12:20:11.978716065Z 2026-06-15 12:20:11.978 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-15T12:20:11.978788602Z 2026-06-15 12:20:11.978 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-15T12:20:12.092882122Z 2026-06-15 12:20:12.092 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-15T12:20:12.099229004Z 2026-06-15 12:20:12.098 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-15T12:20:12.103443203Z 2026-06-15 12:20:12.102 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-15T12:20:12.107986840Z 2026-06-15 12:20:12.107 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-15T12:20:12.122356642Z 2026-06-15 12:20:12.121 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-15T12:20:12.131991487Z 2026-06-15 12:20:12.131 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-15T12:20:12.139399698Z 2026-06-15 12:20:12.138 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n2 solr-node1-6 core_node5 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-15T12:20:12.139427523Z 2026-06-15 12:20:12.138 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n2 solr-node1-6 core_node5 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-15T12:20:12.146988555Z 2026-06-15 12:20:12.146 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n2 solr-node1-6 core_node5 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-15T12:20:12.169191056Z 2026-06-15 12:20:12.168 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-15T12:20:12.170718784Z 2026-06-15 12:20:12.170 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-15T12:20:12.190182490Z 2026-06-15 12:20:12.189 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-15T12:20:12.190438609Z 2026-06-15 12:20:12.189 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-15T12:20:12.190449390Z 2026-06-15 12:20:12.189 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n2/
2026-06-15T12:20:12.191757513Z 2026-06-15 12:20:12.191 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-15T12:20:12.192482081Z 2026-06-15 12:20:12.192 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n2/ has no replicas
2026-06-15T12:20:12.193725061Z 2026-06-15 12:20:12.193 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72126907354775555-core_node5-n_0000000000
2026-06-15T12:20:12.206064304Z 2026-06-15 12:20:12.205 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n2/ shard1
2026-06-15T12:20:12.319909040Z 2026-06-15 12:20:12.319 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:12.324114657Z 2026-06-15 12:20:12.323 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-15T12:20:12.335457923Z 2026-06-15 12:20:12.334 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1450
2026-06-15T12:20:12.436658990Z 2026-06-15 12:20:12.436 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 209] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:12.722071528Z 2026-06-15 12:20:12.721 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:12.757880960Z 2026-06-15 12:20:12.757 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-15T12:20:12.844944812Z 2026-06-15 12:20:12.844 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:12.848610736Z 2026-06-15 12:20:12.848 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2285
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3884889,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-15T12:20:01.67958601Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-15T12:20:02.162433317Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-15T12:20:02.162480468Z It should be set to 65000 to avoid operational disruption.
2026-06-15T12:20:02.162484367Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-15T12:20:02.162785725Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-15T12:20:02.162818489Z
2026-06-15T12:20:02.173609211Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-15T12:20:02.183731167Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-15T12:20:02.235644627Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-15T12:20:02.273483984Z WARNING: A command line option has enabled the Security Manager
2026-06-15T12:20:02.273519470Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-15T12:20:03.643687222Z 2026-06-15 12:20:03.640 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-15T12:20:03.940879528Z 2026-06-15 12:20:03.940 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-15T12:20:03.972269780Z 2026-06-15 12:20:03.971 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-15T12:20:03.979771373Z 2026-06-15 12:20:03.979 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-15T12:20:03.980035651Z 2026-06-15 12:20:03.979 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-15T12:20:03.980242787Z 2026-06-15 12:20:03.979 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-15T12:20:03.980352955Z 2026-06-15 12:20:03.979 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-15T12:20:03.979481474Z
2026-06-15T12:20:03.982990654Z 2026-06-15 12:20:03.982 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-15T12:20:04.004834608Z 2026-06-15 12:20:04.004 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-15T12:20:04.031464382Z 2026-06-15 12:20:04.030 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-15T12:20:04.208315443Z 2026-06-15 12:20:04.207 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-15T12:20:04.228161948Z 2026-06-15 12:20:04.227 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-15T12:20:04.228238131Z 2026-06-15 12:20:04.227 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-15T12:20:04.228436072Z 2026-06-15 12:20:04.227 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-15T12:20:04.339227104Z 2026-06-15 12:20:04.338 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-15T12:20:04.339423802Z 2026-06-15 12:20:04.339 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-15T12:20:04.339477474Z 2026-06-15 12:20:04.339 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-15T12:20:04.375334354Z 2026-06-15 12:20:04.374 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-15T12:20:05.376760224Z 2026-06-15 12:20:05.376 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-15T12:20:05.529010502Z 2026-06-15 12:20:05.528 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-15T12:20:05.530225499Z 2026-06-15 12:20:05.528 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-15T12:20:05.539713251Z 2026-06-15 12:20:05.539 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-15T12:20:05.545142626Z 2026-06-15 12:20:05.544 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-15T12:20:05.549601280Z 2026-06-15 12:20:05.549 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-15T12:20:05.549618585Z 2026-06-15 12:20:05.549 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-15T12:20:05.589306425Z 2026-06-15 12:20:05.588 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-15T12:20:05.598849577Z 2026-06-15 12:20:05.598 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-15T12:20:05.621523336Z 2026-06-15 12:20:05.620 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-15T12:20:05.633542500Z 2026-06-15 12:20:05.633 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-15T12:20:05.644835169Z 2026-06-15 12:20:05.644 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-15T12:20:05.654965355Z 2026-06-15 12:20:05.654 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-15T12:20:05.701413497Z 2026-06-15 12:20:05.700 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-15T12:20:05.973114278Z 2026-06-15 12:20:05.972 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-15T12:20:06.159508467Z 2026-06-15 12:20:06.159 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-15T12:20:06.448323830Z 2026-06-15 12:20:06.447 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-15T12:20:06.601006626Z 2026-06-15 12:20:06.600 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-15T12:20:06.608937991Z 2026-06-15 12:20:06.608 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_15.request.log
2026-06-15T12:20:06.613485139Z 2026-06-15 12:20:06.612 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-15T12:20:06.614729424Z 2026-06-15 12:20:06.614 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4432ms
2026-06-15T12:20:08.956967149Z 2026-06-15 12:20:08.933 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=121
2026-06-15T12:20:09.031195769Z 2026-06-15 12:20:09.030 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=26
2026-06-15T12:20:10.892082755Z 2026-06-15 12:20:10.890 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-15T12:20:11.001592934Z 2026-06-15 12:20:11.001 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-15T12:20:11.146949978Z 2026-06-15 12:20:11.146 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-15T12:20:11.578512956Z 2026-06-15 12:20:11.577 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-15T12:20:11.578626380Z 2026-06-15 12:20:11.577 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-15T12:20:11.607301587Z 2026-06-15 12:20:11.606 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n1], dataDir=[/var/solr/data/test-collection_shard2_replica_n1/data/]
2026-06-15T12:20:12.399624271Z 2026-06-15 12:20:12.399 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-15T12:20:12.457459633Z 2026-06-15 12:20:12.456 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-15T12:20:12.467259633Z 2026-06-15 12:20:12.466 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-15T12:20:12.468519867Z 2026-06-15 12:20:12.467 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-15T12:20:12.468536149Z 2026-06-15 12:20:12.468 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-15T12:20:12.518841125Z 2026-06-15 12:20:12.518 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-15T12:20:12.522969334Z 2026-06-15 12:20:12.522 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-15T12:20:12.525547117Z 2026-06-15 12:20:12.525 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-15T12:20:12.527276711Z 2026-06-15 12:20:12.526 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-15T12:20:12.540328232Z 2026-06-15 12:20:12.539 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-06-15T12:20:12.549554380Z 2026-06-15 12:20:12.549 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-15T12:20:12.557209537Z 2026-06-15 12:20:12.556 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-15T12:20:12.557281853Z 2026-06-15 12:20:12.556 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-15T12:20:12.567450191Z 2026-06-15 12:20:12.566 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-15T12:20:12.575900348Z 2026-06-15 12:20:12.575 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-15T12:20:12.577610415Z 2026-06-15 12:20:12.577 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-15T12:20:12.601365113Z 2026-06-15 12:20:12.600 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-15T12:20:12.601413482Z 2026-06-15 12:20:12.601 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-15T12:20:12.601484964Z 2026-06-15 12:20:12.601 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n1/
2026-06-15T12:20:12.603166442Z 2026-06-15 12:20:12.602 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-15T12:20:12.603782846Z 2026-06-15 12:20:12.603 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ has no replicas
2026-06-15T12:20:12.605093061Z 2026-06-15 12:20:12.604 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72126907354775556-core_node3-n_0000000000
2026-06-15T12:20:12.613337884Z 2026-06-15 12:20:12.612 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ shard2
2026-06-15T12:20:12.722359149Z 2026-06-15 12:20:12.721 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-15T12:20:12.726856489Z 2026-06-15 12:20:12.726 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-15T12:20:12.745487422Z 2026-06-15 12:20:12.745 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1858
2026-06-15T12:20:12.845393921Z 2026-06-15 12:20:12.844 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3885072,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-15T12:20:01.927742403Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 29591 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 66b76b382631: Waiting
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 69a09c96e8bf: Waiting
# 601f2c23751f: Waiting
# 9cdfe5320de8: Waiting
# 0ed719aef393: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# c937aee9a870: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Download complete
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Already exists
# 9d5a0b6cb36c: Already exists
# 49518c28c0b3: Already exists
# 41162acdcc29: Already exists
# 2ed0fd558ca3: Already exists
# 2488415fda1c: Pulling fs layer
# 09511c8fb203: Pulling fs layer
# a0c57fb89c4f: Pulling fs layer
# d0641dcc41d7: Pulling fs layer
# 315722fd9570: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# d0641dcc41d7: Waiting
# 315722fd9570: Waiting
# 4f4fb700ef54: Waiting
# 09511c8fb203: Verifying Checksum
# 09511c8fb203: Download complete
# d0641dcc41d7: Download complete
# a0c57fb89c4f: Verifying Checksum
# a0c57fb89c4f: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 315722fd9570: Verifying Checksum
# 315722fd9570: Download complete
# 2488415fda1c: Verifying Checksum
# 2488415fda1c: Download complete
# 2488415fda1c: Pull complete
# 09511c8fb203: Pull complete
# a0c57fb89c4f: Pull complete
# d0641dcc41d7: Pull complete
# 315722fd9570: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 83b762bb24919b5fda4309a133871323ffb7b79bbe4c936bd229fbdeecd54500
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# b8e0578d26612256047baed44118e14d7d65609a2c73a3c5a111c0a65007441b
# Starting solr-node2 connected to first node's ZooKeeper
# 57f78c6961ed2a879ffdf191e46c9bb800da931c66d29a2bf7c9ba97836b2cb2
# Started solr-node3 connected to first node's ZooKeeper
# 84714353c1f25cf70f0f6ebd22fcd531a837a4905caeb87342abbd3ae9d1fa06
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-15 12:20:10.284; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-15 12:20:10.319; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node2:8984/solr] to [http://solr-node2:8984/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 16596 ms
ok 87 snapshot lifecycle # in 5102 ms
ok 88 snapshot list # in 3578 ms
ok 89 snapshot list via --solr-connection # in 3645 ms
ok 90 snapshot describe # in 3649 ms
ok 91 snapshot describe via --solr-connection # in 3656 ms
ok 92 start solr with ssl # in 17687 ms
ok 93 use different hostname when not checking peer-name # in 26383 ms
ok 94 start solr with ssl and auth # in 15374 ms
ok 95 start solr with client truststore and security manager # in 18422 ms
ok 96 start solr with mTLS needed # in 48026 ms
ok 97 start solr with mTLS wanted # in 37224 ms
ok 98 test keystore reload # in 62377 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 16595 ms
ok 100 stop command for single port # in 19058 ms
ok 101 check stop command doesn't hang # in 249 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8622 ms
not ok 103 deprecated system properties converted to modern properties # in 195627 ms
# (from function `assert_file_contains' in file .gradle/node/node_modules/bats-file/src/file.bash, line 566,
# in test file test/test_start_solr.bats, line 86)
# `assert_file_contains "${SOLR_LOGS_DIR}/solr.log" 'You are passing in deprecated system property disable.config.edit and should upgrade to using solr.api.config.edit.enabled instead.'' failed
# Waiting up to 180 seconds to see Solr running on port 38375 [|] [/] [-] [\] [|]
# Started Solr server on port 38375 (pid=3902658). Happy searching!
#
#
# -- file does not contain regex --
# path : /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/solr.log
# --
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-5
ok 104 start with custom jetty options # in 8732 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 10001 ms
not ok 106 bootstrapping a configset # in 9620 ms
# (from function `config_exists' in file test/bats_helper.bash, line 84,
# in test file test/test_start_solr.bats, line 116)
# `config_exists "techproducts"' failed
# Waiting up to 180 seconds to see Solr running on port 38375 [|] [/] [-] [\] [|]
# Started Solr server on port 38375 (pid=3905353). Happy searching!
#
# WARN - 2026-06-15 12:29:12.279; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-15 12:29:12.283; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-8
ok 107 status detects locally running solr # in 10399 ms
ok 108 status with --solr-url from user # in 8571 ms
ok 109 status with --port from user # in 8437 ms
ok 110 multiple connection options are prevented # in 923 ms
ok 111 status with invalid --solr-url from user # in 1295 ms
ok 112 status with --short format # in 196474 ms
ok 113 searching solr via locally executed streaming expression # in 1889 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1432 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1254 ms
ok 116 variable interpolation # in 1778 ms
ok 117 searching solr without credentials fails with error # in 1221 ms
ok 118 --version returns Solr version # in 789 ms
ok 119 version as direct tool call still runs # in 776 ms
ok 120 short help # in 962 ms
ok 121 short help is inferred # in 813 ms
ok 122 long help # in 860 ms
ok 123 running subcommands with zk is prevented # in 809 ms
ok 124 listing out files # in 2194 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 9954 ms
ok 126 copying files around # in 11541 ms
ok 127 upconfig # in 2638 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1265 ms
ok 129 downconfig # in 1332 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1533 ms
ok 131 env var ZK_HOST is honored # in 3434 ms
ok 132 Cleanup # in 1909 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 15s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/p45d7hflfqyea
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 233 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/233/
No tests ran.
Build Log:
[...truncated 904 lines...]
BUILD FAILED in 29m 42s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/x3hkl6mowsn56
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 232 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/232/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-06-13 12:12:38.767; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-13 12:12:38.771; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-13 12:12:39.220; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-13 12:12:39.220; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 18020 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38917 is already being used by another process (pid: 2551112)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-06-13 12:12:40.406; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-13 12:12:40.410; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 10 post, postlogs and export with basic auth # in 17983 ms
ok 11 package with basic auth # in 12285 ms
ok 12 admin UI static content is excepted from basic auth # in 10532 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 145 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 76 ms
ok 16 compressed response can be decompressed and parsed # in 79 ms
ok 17 setting property # in 11410 ms
ok 18 short form of setting property # in 6545 ms
ok 19 connecting to solr via various connection types # in 10496 ms
ok 20 create for non cloud mode # in 7805 ms
ok 21 create for cloud mode # in 9694 ms
ok 22 multiple connection options are prevented # in 826 ms
ok 23 create collection # in 7692 ms
ok 24 create collection using solrUrl # in 5192 ms
ok 25 create collection using legacy solrUrl # in 5258 ms
ok 26 create collection using Zookeeper # in 5542 ms
ok 27 reject d option with invalid config dir # in 2661 ms
ok 28 accept d option with builtin config # in 5113 ms
ok 29 accept d option with explicit path to config # in 5203 ms
ok 30 accept n option as config name # in 5182 ms
ok 31 allow config reuse when n option specifies same config # in 8892 ms
ok 32 create multisharded collections when s provided # in 5099 ms
ok 33 create replicated collections when rf provided # in 6085 ms
ok 34 can delete collections # in 10903 ms
ok 35 can delete collections with solr-url # in 8387 ms
ok 36 can delete collections with solr-connection # in 7936 ms
ok 37 collection delete also deletes zk config # in 7271 ms
ok 38 deletes accompanying zk config with nondefault name # in 7304 ms
ok 39 delete-config option can opt to leave config in zk # in 7029 ms
not ok 40 start -e cloud works with --jvm-opts # in 222984 ms
# (in test file test/test_example.bats, line 32)
# `solr start -e cloud --no-prompt --jvm-opts "-Dcustom.prop=helloworld"' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Creating Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr
# Cloning /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1 into
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2
#
# Starting up Solr on port 38917 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 38917 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" --jvm-opts "-Dcustom.prop=helloworld"
#
# Waiting up to 180 seconds to see Solr running on port 38917 [|] [/] [-] [\] [|]
# Started Solr server on port 38917 (pid=2565418). Happy searching!
#
#
# Starting up Solr on port 38918 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 38918 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node2/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" -z 127.0.0.1:39917 --jvm-opts "-Dcustom.prop=helloworld"
#
# Waiting up to 180 seconds to see Solr running on port 38918 [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] [|] [/] [-] [\] Still not seeing Solr listening on 38918 after 180 seconds!
# 2026-06-13 12:17:00.976 INFO (main) [] o.e.j.s.Server jetty-12.0.34; built: 2026-04-01T05:45:03.823Z; git: 57e7adb250592bd15f573e7820be9b64632c6637; jvm 21.0.4+7-LTS
#
# ERROR: Did not see Solr at http://localhost:38918/solr come online within 30 seconds!
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
ok 41 start -e cloud works with --no-prompt # in 20933 ms
ok 42 Check export command # in 18971 ms
ok 43 export fails on non cloud mode # in 9676 ms
Waiting for Tika Server to be ready on port 38922
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 44 using curl to extract a single pdf file # in 16574 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to extract content from pdf # in 16538 ms
WARNING: Docker not available; bypassing test.
ok 46 using the bin/solr post tool to crawl web site # in 16607 ms
Stopping Tika Server container
ok 47 healthcheck on cloud solr # in 10193 ms
ok 48 healthcheck on remote cloud solr # in 9745 ms
ok 49 healthcheck errors on standalone solr # in 11089 ms
ok 50 solr help flag prints help # in 722 ms
ok 51 solr with no flags prints help # in 711 ms
ok 52 start help flag prints help # in 141 ms
ok 53 start h flag prints help # in 139 ms
ok 54 stop help flag prints help # in 135 ms
ok 55 restart help flag prints help # in 143 ms
ok 56 status help flag prints help # in 767 ms
ok 57 healthcheck help flag prints help # in 808 ms
ok 58 create help flag prints help # in 751 ms
ok 59 delete help flag prints help # in 792 ms
ok 60 zk help flag prints help # in 771 ms
ok 61 auth help flag prints help # in 728 ms
ok 62 assert help flag prints help # in 741 ms
ok 63 post help flag prints help # in 758 ms
ok 64 SQL Module # in 16222 ms
ok 65 icu collation in analysis-extras module # in 12558 ms
ok 66 Check lifecycle of sentiment classification # in 22774 ms
ok 67 lifecycle of package # in 8903 ms
ok 68 deploying and undeploying a collection level package # in 13494 ms
ok 69 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 70 Affinity placement plugin using sysprop # in 15214 ms
ok 71 Random placement plugin using ENV # in 15576 ms
ok 72 Check help command # in 2284 ms
ok 73 basic post with a type specified # in 5661 ms
ok 74 basic post with collection instead of url parameter # in 4078 ms
ok 75 basic post with solr-url and collection # in 3902 ms
ok 76 basic post with -s (solr-connection short form) and collection # in 3883 ms
ok 77 basic post WITHOUT a type specified # in 11467 ms
ok 78 crawling a directory as a dry-run # in 774 ms
ok 79 crawling a directory # in 4180 ms
ok 80 skipcommit and optimize and delete # in 3735 ms
ok 81 args mode # in 6753 ms
ok 82 stdin mode # in 3532 ms
ok 83 verbose echo the Solr response # in 3828 ms
ok 84 post solr log into solr via cli # in 9513 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-06-13T12:26:15.812111800Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-13T12:26:15.812158633Z It should be set to 65000 to avoid operational disruption.
2026-06-13T12:26:15.812163635Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-13T12:26:15.812446326Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-13T12:26:15.812466044Z
2026-06-13T12:26:15.822239326Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-13T12:26:15.829864968Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-13T12:26:15.878473229Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-13T12:26:15.909319018Z WARNING: A command line option has enabled the Security Manager
2026-06-13T12:26:15.909350961Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-13T12:26:17.113130016Z 2026-06-13 12:26:17.109 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-13T12:26:17.378699794Z 2026-06-13 12:26:17.378 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-13T12:26:17.404040293Z 2026-06-13 12:26:17.403 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-13T12:26:17.411393076Z 2026-06-13 12:26:17.410 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-13T12:26:17.411422536Z 2026-06-13 12:26:17.411 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-06-13T12:26:17.411608505Z 2026-06-13 12:26:17.411 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-13T12:26:17.411745356Z 2026-06-13 12:26:17.411 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-13T12:26:17.411161157Z
2026-06-13T12:26:17.414493270Z 2026-06-13 12:26:17.414 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-13T12:26:17.437205336Z 2026-06-13 12:26:17.436 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-13T12:26:17.451796751Z 2026-06-13 12:26:17.451 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-13T12:26:17.451829311Z 2026-06-13 12:26:17.451 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-13T12:26:17.489839676Z 2026-06-13 12:26:17.489 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-13T12:26:18.169651239Z 2026-06-13 12:26:18.169 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-13T12:26:18.404483391Z 2026-06-13 12:26:18.403 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-13T12:26:18.409788599Z 2026-06-13 12:26:18.409 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-06-13T12:26:18.409839285Z 2026-06-13 12:26:18.409 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-06-13T12:26:18.412444193Z 2026-06-13 12:26:18.412 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-06-13T12:26:18.412517098Z 2026-06-13 12:26:18.412 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-06-13T12:26:18.448122099Z 2026-06-13 12:26:18.447 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-06-13T12:26:18.914770493Z 2026-06-13 12:26:18.914 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-06-13T12:26:18.928912269Z 2026-06-13 12:26:18.928 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-13T12:26:19.065106389Z 2026-06-13 12:26:19.064 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-13T12:26:19.091188339Z 2026-06-13 12:26:19.090 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-13T12:26:19.091210261Z 2026-06-13 12:26:19.090 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-13T12:26:19.216653853Z 2026-06-13 12:26:19.216 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-13T12:26:19.238923024Z 2026-06-13 12:26:19.238 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-13T12:26:19.250599445Z 2026-06-13 12:26:19.250 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-06-13T12:26:19.257083584Z 2026-06-13 12:26:19.256 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72115607593287680-solr-node1:8983_solr-n_0000000000) starting
2026-06-13T12:26:19.410887243Z 2026-06-13 12:26:19.410 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-06-13T12:26:19.411740596Z 2026-06-13 12:26:19.411 INFO (OverseerStateUpdate-72115607593287680-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-06-13T12:26:19.421500909Z 2026-06-13 12:26:19.421 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-13T12:26:19.588153269Z 2026-06-13 12:26:19.587 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-13T12:26:20.052548266Z 2026-06-13 12:26:20.050 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-13T12:26:20.243930959Z 2026-06-13 12:26:20.243 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-13T12:26:20.536938051Z 2026-06-13 12:26:20.536 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-13T12:26:20.693490588Z 2026-06-13 12:26:20.692 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-13T12:26:20.700999432Z 2026-06-13 12:26:20.700 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_13.request.log
2026-06-13T12:26:20.777713866Z 2026-06-13 12:26:20.776 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-06-13T12:26:20.778789648Z 2026-06-13 12:26:20.778 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4950ms
2026-06-13T12:26:21.135694375Z 2026-06-13 12:26:21.134 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=46
2026-06-13T12:26:21.415489011Z 2026-06-13 12:26:21.414 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-06-13T12:26:24.635339439Z 2026-06-13 12:26:24.634 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-13T12:26:24.946427915Z 2026-06-13 12:26:24.945 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-13T12:26:28.809414639Z 2026-06-13 12:26:28.808 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-06-13T12:26:28.838529541Z 2026-06-13 12:26:28.838 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-06-13T12:26:28.878279519Z 2026-06-13 12:26:28.877 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-06-13T12:26:29.517300487Z 2026-06-13 12:26:29.516 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-06-13T12:26:29.543668422Z 2026-06-13 12:26:29.543 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-06-13T12:26:29.549615429Z 2026-06-13 12:26:29.549 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:29.595652029Z 2026-06-13 12:26:29.594 INFO (OverseerStateUpdate-72115607593287680-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-13T12:26:29.595675980Z "core":"test-collection_shard3_replica_n1",
2026-06-13T12:26:29.595679808Z "node_name":"solr-node1:8983_solr",
2026-06-13T12:26:29.595682663Z "base_url":"http://solr-node1:8983/solr",
2026-06-13T12:26:29.595685424Z "collection":"test-collection",
2026-06-13T12:26:29.595688096Z "shard":"shard3",
2026-06-13T12:26:29.595690777Z "state":"down",
2026-06-13T12:26:29.595693342Z "type":"NRT",
2026-06-13T12:26:29.595695890Z "operation":"ADDREPLICA",
2026-06-13T12:26:29.595698507Z "waitForFinalState":"false"}
2026-06-13T12:26:29.603611797Z 2026-06-13 12:26:29.602 INFO (OverseerStateUpdate-72115607593287680-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-13T12:26:29.603637906Z "core":"test-collection_shard2_replica_n2",
2026-06-13T12:26:29.603642676Z "node_name":"solr-node3:8985_solr",
2026-06-13T12:26:29.603646812Z "base_url":"http://solr-node3:8985/solr",
2026-06-13T12:26:29.603650914Z "collection":"test-collection",
2026-06-13T12:26:29.603654899Z "shard":"shard2",
2026-06-13T12:26:29.603658959Z "state":"down",
2026-06-13T12:26:29.603663051Z "type":"NRT",
2026-06-13T12:26:29.603666959Z "operation":"ADDREPLICA",
2026-06-13T12:26:29.603670972Z "waitForFinalState":"false"}
2026-06-13T12:26:29.611897926Z 2026-06-13 12:26:29.611 INFO (OverseerStateUpdate-72115607593287680-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-06-13T12:26:29.611922495Z "core":"test-collection_shard1_replica_n4",
2026-06-13T12:26:29.611926126Z "node_name":"solr-node2:8984_solr",
2026-06-13T12:26:29.611943426Z "base_url":"http://solr-node2:8984/solr",
2026-06-13T12:26:29.611945744Z "collection":"test-collection",
2026-06-13T12:26:29.611947796Z "shard":"shard1",
2026-06-13T12:26:29.611949846Z "state":"down",
2026-06-13T12:26:29.611951780Z "type":"NRT",
2026-06-13T12:26:29.611953811Z "operation":"ADDREPLICA",
2026-06-13T12:26:29.611955826Z "waitForFinalState":"false"}
2026-06-13T12:26:29.722535842Z 2026-06-13 12:26:29.721 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:29.803905767Z 2026-06-13 12:26:29.802 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-06-13T12:26:29.873439070Z 2026-06-13 12:26:29.872 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-13T12:26:29.950363169Z 2026-06-13 12:26:29.949 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-13T12:26:30.989032981Z 2026-06-13 12:26:30.988 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-13T12:26:30.989116529Z 2026-06-13 12:26:30.988 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-06-13T12:26:31.018760386Z 2026-06-13 12:26:31.018 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-06-13T12:26:31.799522025Z 2026-06-13 12:26:31.798 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-13T12:26:31.838954549Z 2026-06-13 12:26:31.838 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-13T12:26:31.846116100Z 2026-06-13 12:26:31.845 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-13T12:26:31.847101766Z 2026-06-13 12:26:31.846 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-13T12:26:31.847934274Z 2026-06-13 12:26:31.846 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-13T12:26:31.947116503Z 2026-06-13 12:26:31.946 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-13T12:26:31.951772874Z 2026-06-13 12:26:31.951 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-13T12:26:31.955927126Z 2026-06-13 12:26:31.955 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-13T12:26:31.958320316Z 2026-06-13 12:26:31.957 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-13T12:26:31.979738616Z 2026-06-13 12:26:31.979 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-13T12:26:31.988679970Z 2026-06-13 12:26:31.988 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-13T12:26:31.999000915Z 2026-06-13 12:26:31.998 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-13T12:26:31.999067980Z 2026-06-13 12:26:31.998 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-13T12:26:32.007005166Z 2026-06-13 12:26:32.006 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-13T12:26:32.019011079Z 2026-06-13 12:26:32.018 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-06-13T12:26:32.023349248Z 2026-06-13 12:26:32.022 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-06-13T12:26:32.038824136Z 2026-06-13 12:26:32.038 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-13T12:26:32.038849409Z 2026-06-13 12:26:32.038 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-13T12:26:32.038864137Z 2026-06-13 12:26:32.038 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-06-13T12:26:32.040652638Z 2026-06-13 12:26:32.040 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-13T12:26:32.041447611Z 2026-06-13 12:26:32.041 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-06-13T12:26:32.043463416Z 2026-06-13 12:26:32.043 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72115607593287680-core_node3-n_0000000000
2026-06-13T12:26:32.050731006Z 2026-06-13 12:26:32.050 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-06-13T12:26:32.156831423Z 2026-06-13 12:26:32.156 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:32.160082200Z 2026-06-13 12:26:32.159 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-13T12:26:32.184212873Z 2026-06-13 12:26:32.183 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2382
2026-06-13T12:26:32.283687621Z 2026-06-13 12:26:32.283 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2583488,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-13T12:26:15.593210577Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-06-13T12:26:21.723760899Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-13T12:26:21.723781999Z It should be set to 65000 to avoid operational disruption.
2026-06-13T12:26:21.723784882Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-13T12:26:21.724131595Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-13T12:26:21.724147513Z
2026-06-13T12:26:21.733357628Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-13T12:26:21.741442665Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-13T12:26:21.782426913Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-13T12:26:21.817724382Z WARNING: A command line option has enabled the Security Manager
2026-06-13T12:26:21.817747419Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-13T12:26:22.950920254Z 2026-06-13 12:26:22.947 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-13T12:26:23.214228859Z 2026-06-13 12:26:23.213 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-13T12:26:23.239916627Z 2026-06-13 12:26:23.239 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-13T12:26:23.247618363Z 2026-06-13 12:26:23.247 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-13T12:26:23.247741300Z 2026-06-13 12:26:23.247 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-06-13T12:26:23.247874949Z 2026-06-13 12:26:23.247 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-13T12:26:23.247995946Z 2026-06-13 12:26:23.247 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-13T12:26:23.247334625Z
2026-06-13T12:26:23.250720247Z 2026-06-13 12:26:23.250 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-13T12:26:23.273226061Z 2026-06-13 12:26:23.272 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-13T12:26:23.296222660Z 2026-06-13 12:26:23.295 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-13T12:26:23.434848186Z 2026-06-13 12:26:23.434 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-13T12:26:23.454067461Z 2026-06-13 12:26:23.453 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-13T12:26:23.454257793Z 2026-06-13 12:26:23.453 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-13T12:26:23.454375313Z 2026-06-13 12:26:23.453 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-13T12:26:23.565774709Z 2026-06-13 12:26:23.565 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-13T12:26:23.565970330Z 2026-06-13 12:26:23.565 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-13T12:26:23.566138444Z 2026-06-13 12:26:23.565 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-13T12:26:23.602310927Z 2026-06-13 12:26:23.601 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-13T12:26:24.326165072Z 2026-06-13 12:26:24.325 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-13T12:26:24.462425855Z 2026-06-13 12:26:24.461 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-13T12:26:24.463859543Z 2026-06-13 12:26:24.462 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-13T12:26:24.488158339Z 2026-06-13 12:26:24.487 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-13T12:26:24.496246564Z 2026-06-13 12:26:24.495 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-13T12:26:24.511139815Z 2026-06-13 12:26:24.510 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-13T12:26:24.511247972Z 2026-06-13 12:26:24.510 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-13T12:26:24.557234899Z 2026-06-13 12:26:24.556 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-13T12:26:24.566268763Z 2026-06-13 12:26:24.565 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-06-13T12:26:24.590026071Z 2026-06-13 12:26:24.589 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-13T12:26:24.605021654Z 2026-06-13 12:26:24.604 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-06-13T12:26:24.622612019Z 2026-06-13 12:26:24.622 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-06-13T12:26:24.638093394Z 2026-06-13 12:26:24.637 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-06-13T12:26:24.683226732Z 2026-06-13 12:26:24.682 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-13T12:26:24.946777565Z 2026-06-13 12:26:24.946 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-13T12:26:24.964228457Z 2026-06-13 12:26:24.963 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-13T12:26:25.152481995Z 2026-06-13 12:26:25.151 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-13T12:26:25.430121259Z 2026-06-13 12:26:25.429 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-13T12:26:25.578053182Z 2026-06-13 12:26:25.577 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-13T12:26:25.584587657Z 2026-06-13 12:26:25.584 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_13.request.log
2026-06-13T12:26:25.588303580Z 2026-06-13 12:26:25.587 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-06-13T12:26:25.589748679Z 2026-06-13 12:26:25.589 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3849ms
2026-06-13T12:26:25.797965176Z 2026-06-13 12:26:25.797 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=73
2026-06-13T12:26:25.857627768Z 2026-06-13 12:26:25.857 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-06-13T12:26:29.799064319Z 2026-06-13 12:26:29.798 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-06-13T12:26:29.856768664Z 2026-06-13 12:26:29.856 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-13T12:26:29.918207149Z 2026-06-13 12:26:29.917 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-13T12:26:30.738686444Z 2026-06-13 12:26:30.737 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-13T12:26:30.738709600Z 2026-06-13 12:26:30.738 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-06-13T12:26:30.757266713Z 2026-06-13 12:26:30.756 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-06-13T12:26:31.301494387Z 2026-06-13 12:26:31.300 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-13T12:26:31.322221596Z 2026-06-13 12:26:31.321 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-13T12:26:31.326099508Z 2026-06-13 12:26:31.325 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-13T12:26:31.327582023Z 2026-06-13 12:26:31.326 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-13T12:26:31.327620937Z 2026-06-13 12:26:31.327 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-13T12:26:31.373495160Z 2026-06-13 12:26:31.373 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-13T12:26:31.377025848Z 2026-06-13 12:26:31.376 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-13T12:26:31.379512595Z 2026-06-13 12:26:31.379 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-13T12:26:31.381435501Z 2026-06-13 12:26:31.381 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-13T12:26:31.397864069Z 2026-06-13 12:26:31.397 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-06-13T12:26:31.407967158Z 2026-06-13 12:26:31.407 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-13T12:26:31.414550414Z 2026-06-13 12:26:31.414 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-13T12:26:31.414575002Z 2026-06-13 12:26:31.414 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-13T12:26:31.425155526Z 2026-06-13 12:26:31.424 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-13T12:26:31.433443814Z 2026-06-13 12:26:31.432 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-06-13T12:26:31.435411720Z 2026-06-13 12:26:31.434 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-06-13T12:26:31.454340146Z 2026-06-13 12:26:31.453 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-13T12:26:31.454370274Z 2026-06-13 12:26:31.454 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-13T12:26:31.454376136Z 2026-06-13 12:26:31.454 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-06-13T12:26:31.456353872Z 2026-06-13 12:26:31.455 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-13T12:26:31.459874458Z 2026-06-13 12:26:31.459 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-06-13T12:26:31.461429579Z 2026-06-13 12:26:31.461 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72115607593287683-core_node6-n_0000000000
2026-06-13T12:26:31.560955818Z 2026-06-13 12:26:31.560 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-06-13T12:26:31.763236446Z 2026-06-13 12:26:31.762 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:31.767081585Z 2026-06-13 12:26:31.766 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-13T12:26:31.776745951Z 2026-06-13 12:26:31.776 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1979
2026-06-13T12:26:31.879712130Z 2026-06-13 12:26:31.879 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:32.157006972Z 2026-06-13 12:26:32.156 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:32.283684919Z 2026-06-13 12:26:32.283 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2583883,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-13T12:26:21.50631586Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-06-13T12:26:21.954026113Z *** [WARN] *** Your open file limit is currently 1024.
2026-06-13T12:26:21.954061448Z It should be set to 65000 to avoid operational disruption.
2026-06-13T12:26:21.954064461Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-06-13T12:26:21.954347202Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-06-13T12:26:21.954352142Z
2026-06-13T12:26:21.964884524Z Java 17 detected. Enabled workaround for SOLR-16463
2026-06-13T12:26:21.972698500Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-06-13T12:26:22.019786705Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-06-13T12:26:22.052792407Z WARNING: A command line option has enabled the Security Manager
2026-06-13T12:26:22.052833613Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-06-13T12:26:23.087005764Z 2026-06-13 12:26:23.083 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-06-13T12:26:23.383290909Z 2026-06-13 12:26:23.382 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-06-13T12:26:23.409011311Z 2026-06-13 12:26:23.408 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-06-13T12:26:23.416293946Z 2026-06-13 12:26:23.415 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-06-13T12:26:23.416343533Z 2026-06-13 12:26:23.415 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-06-13T12:26:23.416451062Z 2026-06-13 12:26:23.415 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-06-13T12:26:23.416588884Z 2026-06-13 12:26:23.416 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-06-13T12:26:23.416049544Z
2026-06-13T12:26:23.419281310Z 2026-06-13 12:26:23.418 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-06-13T12:26:23.444047420Z 2026-06-13 12:26:23.443 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-06-13T12:26:23.472541613Z 2026-06-13 12:26:23.471 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-13T12:26:23.623283779Z 2026-06-13 12:26:23.622 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-06-13T12:26:23.645968706Z 2026-06-13 12:26:23.645 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-13T12:26:23.646016337Z 2026-06-13 12:26:23.645 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-13T12:26:23.646138250Z 2026-06-13 12:26:23.645 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-06-13T12:26:23.756793399Z 2026-06-13 12:26:23.756 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-06-13T12:26:23.757052023Z 2026-06-13 12:26:23.756 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-06-13T12:26:23.757144874Z 2026-06-13 12:26:23.756 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-06-13T12:26:23.792789535Z 2026-06-13 12:26:23.792 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-06-13T12:26:24.634040199Z 2026-06-13 12:26:24.633 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-06-13T12:26:24.799735625Z 2026-06-13 12:26:24.799 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-06-13T12:26:24.801162187Z 2026-06-13 12:26:24.799 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-06-13T12:26:24.812986972Z 2026-06-13 12:26:24.812 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-13T12:26:24.819283347Z 2026-06-13 12:26:24.818 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-06-13T12:26:24.822475386Z 2026-06-13 12:26:24.822 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-06-13T12:26:24.822762340Z 2026-06-13 12:26:24.822 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-06-13T12:26:24.858532440Z 2026-06-13 12:26:24.857 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-06-13T12:26:24.870770033Z 2026-06-13 12:26:24.870 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-06-13T12:26:24.899254972Z 2026-06-13 12:26:24.898 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-06-13T12:26:24.919681667Z 2026-06-13 12:26:24.919 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-06-13T12:26:24.933772874Z 2026-06-13 12:26:24.933 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-06-13T12:26:24.947990889Z 2026-06-13 12:26:24.947 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-06-13T12:26:25.001020377Z 2026-06-13 12:26:25.000 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-06-13T12:26:25.284173790Z 2026-06-13 12:26:25.283 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-06-13T12:26:25.453077181Z 2026-06-13 12:26:25.452 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-06-13T12:26:25.714340037Z 2026-06-13 12:26:25.713 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-13T12:26:25.844362706Z 2026-06-13 12:26:25.843 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-06-13T12:26:25.849731210Z 2026-06-13 12:26:25.849 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_06_13.request.log
2026-06-13T12:26:25.853103965Z 2026-06-13 12:26:25.852 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-06-13T12:26:25.853889686Z 2026-06-13 12:26:25.853 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3882ms
2026-06-13T12:26:27.555058978Z 2026-06-13 12:26:27.548 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=117
2026-06-13T12:26:27.633101351Z 2026-06-13 12:26:27.632 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=24
2026-06-13T12:26:29.477233061Z 2026-06-13 12:26:29.476 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=10
2026-06-13T12:26:29.801271128Z 2026-06-13 12:26:29.799 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-06-13T12:26:29.876389237Z 2026-06-13 12:26:29.875 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-06-13T12:26:29.980994171Z 2026-06-13 12:26:29.980 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-06-13T12:26:30.892786325Z 2026-06-13 12:26:30.892 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-06-13T12:26:30.892885803Z 2026-06-13 12:26:30.892 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-06-13T12:26:30.916444721Z 2026-06-13 12:26:30.915 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-06-13T12:26:31.431369827Z 2026-06-13 12:26:31.430 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-06-13T12:26:31.457383817Z 2026-06-13 12:26:31.456 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-06-13T12:26:31.467068655Z 2026-06-13 12:26:31.466 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-06-13T12:26:31.468428190Z 2026-06-13 12:26:31.467 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-06-13T12:26:31.468651274Z 2026-06-13 12:26:31.468 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-06-13T12:26:31.558776494Z 2026-06-13 12:26:31.558 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-06-13T12:26:31.562694380Z 2026-06-13 12:26:31.562 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-06-13T12:26:31.565540116Z 2026-06-13 12:26:31.565 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-06-13T12:26:31.568258495Z 2026-06-13 12:26:31.567 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-06-13T12:26:31.583845122Z 2026-06-13 12:26:31.583 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-06-13T12:26:31.591416615Z 2026-06-13 12:26:31.590 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-06-13T12:26:31.598015381Z 2026-06-13 12:26:31.597 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-06-13T12:26:31.598044620Z 2026-06-13 12:26:31.597 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-06-13T12:26:31.604517935Z 2026-06-13 12:26:31.604 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-06-13T12:26:31.618380680Z 2026-06-13 12:26:31.617 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-06-13T12:26:31.620419001Z 2026-06-13 12:26:31.620 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-06-13T12:26:31.638303606Z 2026-06-13 12:26:31.637 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-06-13T12:26:31.638536972Z 2026-06-13 12:26:31.638 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-06-13T12:26:31.638622637Z 2026-06-13 12:26:31.638 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-06-13T12:26:31.641007187Z 2026-06-13 12:26:31.640 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-06-13T12:26:31.641968154Z 2026-06-13 12:26:31.641 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-06-13T12:26:31.643843347Z 2026-06-13 12:26:31.643 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72115607593287684-core_node5-n_0000000000
2026-06-13T12:26:31.656593183Z 2026-06-13 12:26:31.656 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-06-13T12:26:31.763348498Z 2026-06-13 12:26:31.762 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:31.767950534Z 2026-06-13 12:26:31.767 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-06-13T12:26:31.776809863Z 2026-06-13 12:26:31.776 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1977
2026-06-13T12:26:31.879552862Z 2026-06-13 12:26:31.879 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:32.157273650Z 2026-06-13 12:26:32.156 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:32.226398236Z 2026-06-13 12:26:32.225 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-06-13T12:26:32.283694010Z 2026-06-13 12:26:32.283 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-06-13T12:26:32.287653089Z 2026-06-13 12:26:32.287 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2794
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2584064,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-06-13T12:26:21.727617904Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 85 Docker SolrCloud rolling upgrade # in 28065 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 66b76b382631: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 69a09c96e8bf: Waiting
# 3aa9120e46f5: Waiting
# 0ed719aef393: Waiting
# 4f4fb700ef54: Waiting
# 601f2c23751f: Waiting
# cb0efb96dabd: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Already exists
# 9d5a0b6cb36c: Already exists
# 49518c28c0b3: Already exists
# 41162acdcc29: Already exists
# 2ed0fd558ca3: Already exists
# 2488415fda1c: Pulling fs layer
# 09511c8fb203: Pulling fs layer
# a0c57fb89c4f: Pulling fs layer
# d0641dcc41d7: Pulling fs layer
# 315722fd9570: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# d0641dcc41d7: Waiting
# 315722fd9570: Waiting
# 4f4fb700ef54: Waiting
# 09511c8fb203: Verifying Checksum
# 09511c8fb203: Download complete
# a0c57fb89c4f: Verifying Checksum
# a0c57fb89c4f: Download complete
# d0641dcc41d7: Download complete
# 315722fd9570: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 2488415fda1c: Verifying Checksum
# 2488415fda1c: Download complete
# 2488415fda1c: Pull complete
# 09511c8fb203: Pull complete
# a0c57fb89c4f: Pull complete
# d0641dcc41d7: Pull complete
# 315722fd9570: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:fb25f7a7190e3027ade20e97708a8665d41ab0e9ea9285fa242b2c83b0e5f43c
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 180b9143cc2020eb54b4a1ec50f157ab819ee0bd8878fab0fcb4bfd40db0d184
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# e1b0d29d3d30e577dbf76dbbc02783720cc5ff12daf088ef228db86459cdc32f
# Starting solr-node2 connected to first node's ZooKeeper
# 1135ee30140a867bc6da1cb87db5a6251a2d982cafb8e2a24d30a274ea9fb688
# Started solr-node3 connected to first node's ZooKeeper
# 81754931140e2839a256f6daaebc391a83628d8f6d7d099de1e3f110e873cbf2
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-06-13 12:26:28.917; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-13 12:26:28.952; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 86 allowPaths - backup # in 15343 ms
ok 87 snapshot lifecycle # in 4821 ms
ok 88 snapshot list # in 3339 ms
ok 89 snapshot list via --solr-connection # in 3317 ms
ok 90 snapshot describe # in 3332 ms
ok 91 snapshot describe via --solr-connection # in 3466 ms
ok 92 start solr with ssl # in 16182 ms
ok 93 use different hostname when not checking peer-name # in 24822 ms
ok 94 start solr with ssl and auth # in 14735 ms
ok 95 start solr with client truststore and security manager # in 17064 ms
ok 96 start solr with mTLS needed # in 46409 ms
ok 97 start solr with mTLS wanted # in 35595 ms
ok 98 test keystore reload # in 59463 ms
ok 99 SOLR-11740 check 'solr stop' connection # in 16152 ms
ok 100 stop command for single port # in 18393 ms
ok 101 check stop command doesn't hang # in 224 ms
ok 102 SOLR-16976 solr starts with remote JMX enabled # in 8194 ms
not ok 103 deprecated system properties converted to modern properties # in 195110 ms
# (from function `assert_file_contains' in file .gradle/node/node_modules/bats-file/src/file.bash, line 566,
# in test file test/test_start_solr.bats, line 86)
# `assert_file_contains "${SOLR_LOGS_DIR}/solr.log" 'You are passing in deprecated system property disable.config.edit and should upgrade to using solr.api.config.edit.enabled instead.'' failed
# Waiting up to 180 seconds to see Solr running on port 38917 [|] [/] [-] [\] [|]
# Started Solr server on port 38917 (pid=2601704). Happy searching!
#
#
# -- file does not contain regex --
# path : /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/solr.log
# --
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-5
ok 104 start with custom jetty options # in 8299 ms
ok 105 -c flag prints no-op warning and still starts in cloud mode # in 9625 ms
not ok 106 bootstrapping a configset # in 9482 ms
# (from function `config_exists' in file test/bats_helper.bash, line 84,
# in test file test/test_start_solr.bats, line 116)
# `config_exists "techproducts"' failed
# Waiting up to 180 seconds to see Solr running on port 38917 [|] [/] [-] [\] [|]
# Started Solr server on port 38917 (pid=2606706). Happy searching!
#
# WARN - 2026-06-13 12:35:14.529; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-06-13 12:35:14.532; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #8 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-8
ok 107 status detects locally running solr # in 9595 ms
ok 108 status with --solr-url from user # in 7886 ms
ok 109 status with --port from user # in 8050 ms
ok 110 multiple connection options are prevented # in 835 ms
ok 111 status with invalid --solr-url from user # in 1282 ms
ok 112 status with --short format # in 196112 ms
ok 113 searching solr via locally executed streaming expression # in 1732 ms
ok 114 searching solr via remotely executed streaming expression by solr-url # in 1255 ms
ok 115 searching solr via remotely executed streaming expression by solr connection # in 1134 ms
ok 116 variable interpolation # in 1702 ms
ok 117 searching solr without credentials fails with error # in 1112 ms
ok 118 --version returns Solr version # in 775 ms
ok 119 version as direct tool call still runs # in 836 ms
ok 120 short help # in 789 ms
ok 121 short help is inferred # in 766 ms
ok 122 long help # in 748 ms
ok 123 running subcommands with zk is prevented # in 720 ms
ok 124 listing out files # in 2115 ms
ok 125 connecting to solr via various solr urls and zk hosts # in 9122 ms
ok 126 copying files around # in 10982 ms
ok 127 upconfig # in 2510 ms
ok 128 SOLR-12429 test upconfig fails with symlink # in 1191 ms
ok 129 downconfig # in 1185 ms
ok 130 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1412 ms
ok 131 env var ZK_HOST is honored # in 3159 ms
ok 132 Cleanup # in 1994 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 33m 10s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/voqti3c7huzjs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 231 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/231/
No tests ran.
Build Log:
[...truncated 984 lines...]
BUILD FAILED in 31m 1s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/g235fcefimeo6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 230 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/230/
No tests ran.
Build Log:
[...truncated 906 lines...]
BUILD FAILED in 29m 48s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/nuesiyjybwrtq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 229 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/229/
No tests ran.
Build Log:
[...truncated 981 lines...]
BUILD FAILED in 29m 47s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/sw7nn4booqalm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 228 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/228/
No tests ran.
Build Log:
[...truncated 983 lines...]
BUILD FAILED in 29m 26s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/abvfyr5262t6u
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 227 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/227/
No tests ran.
Build Log:
[...truncated 881 lines...]
BUILD FAILED in 31m
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/nvpfkfes52mba
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 226 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/226/
No tests ran.
Build Log:
[...truncated 881 lines...]
BUILD FAILED in 29m 26s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/riu2i7k3eqheq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 225 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/225/
No tests ran.
Build Log:
[...truncated 956 lines...]
BUILD FAILED in 30m 6s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/kqemboxba4uyo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 224 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/224/
No tests ran.
Build Log:
[...truncated 959 lines...]
BUILD FAILED in 29m 12s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/uku3dms3w4lqm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 223 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/223/
No tests ran.
Build Log:
[...truncated 958 lines...]
BUILD FAILED in 30m 28s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/4p7z5vbdyxjge
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 222 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/222/
No tests ran.
Build Log:
[...truncated 957 lines...]
BUILD FAILED in 31m 30s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/hoojki5rlfxb4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 221 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/221/
No tests ran.
Build Log:
[...truncated 877 lines...]
BUILD FAILED in 29m 13s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/iifzr3kingji6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 220 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/220/
No tests ran.
Build Log:
[...truncated 974 lines...]
BUILD FAILED in 39m 23s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/uvaxb4xzctoba
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 219 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/219/
No tests ran.
Build Log:
[...truncated 874 lines...]
BUILD FAILED in 28m 5s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/6yhhy4azwsc42
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 218 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/218/
No tests ran.
Build Log:
[...truncated 881 lines...]
BUILD FAILED in 30m 4s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/giq742fit5llg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 217 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/217/
No tests ran.
Build Log:
[...truncated 375 lines...]
# ERROR: Solr is not running on url http://localhost:38713 after 5 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/4-2
# Last output:
# Waiting up to 180 seconds to see Solr running on port 38713 [|] [/] [-] [\] [|]
# Started Solr server on port 38713 (pid=85186). Happy searching!
#
#
ok 5 auth rejects block-unknown option with invalid boolean # in 1828 ms
ok 6 auth rejects update-include-file-only option with invalid boolean # in 1867 ms
not ok 7 auth enable/disable lifecycle # in 20591 ms
# (in test file test/test_auth.bats, line 40)
# `solr auth enable --type basicAuth --credentials name:password' failed
# Waiting up to 180 seconds to see Solr running on port 38713 [|] [/] [-] [\] [|]
# Started Solr server on port 38713 (pid=87460). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
# Couldn't get ZooKeeper host. Please make sure Solr is running in cloud mode, or a zk-host has been passed in.
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Server refused connection at: http://localhost:38713/solr/admin/info/system
not ok 8 enable auth connects via zookeeper # in 1310 ms
# (in test file test/test_auth.bats, line 53)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 9 create, config, api, and delete with basic auth # in 16695 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-26 16:31:45.427; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:31:45.431; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 10 post, postlogs and export with basic auth # in 16748 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/10-2
# Last output:
# WARN - 2026-05-26 16:32:02.198; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:32:02.202; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 11 package with basic auth # in 16636 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/11-3
# Last output:
# WARN - 2026-05-26 16:32:18.864; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:32:18.867; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 12 admin UI static content is excepted from basic auth # in 16664 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/12-4
# Last output:
# WARN - 2026-05-26 16:32:35.548; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:32:35.552; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 13 setup_file failed
# (from function `setup_file' in test file test/test_bats.bats, line 26)
# `solr start --verbose' failed
# Using Solr root directory: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT
# Using Java: /home/jenkins/tools/java/latest21/bin/java
# openjdk version "21.0.4" 2024-07-16 LTS
# OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
# OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
not ok 14 setup_file failed
# (from function `setup_file' in test file test/test_compression.bats, line 22)
# `solr start -e films' failed
#
# Starting up Solr on port 38713 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 38713 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 38713 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test # at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/-
not ok 17 setup_file failed
# (from function `setup_file' in test file test/test_config.bats, line 22)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
not ok 20 create for non cloud mode # in 1452 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 34)
# `assert_output --partial "Created new core 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created new core 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/20-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 21 create for cloud mode # in 1454 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 40)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/21-2
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
ok 22 multiple connection options are prevented # in 851 ms
not ok 23 setup_file failed
# (from function `setup_file' in test file test/test_create_collection.bats, line 22)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
not ok 34 setup_file failed
# (from function `setup_file' in test file test/test_delete_collection.bats, line 22)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
not ok 39 start -e cloud works with --jvm-opts # in 1479 ms
# (in test file test/test_example.bats, line 32)
# `solr start -e cloud --no-prompt --jvm-opts "-Dcustom.prop=helloworld"' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Creating Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr
# Cloning /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1 into
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2
#
# Starting up Solr on port 38713 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 38713 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" --jvm-opts "-Dcustom.prop=helloworld"
#
#
# ERROR: Port 38713 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/39-1
not ok 40 start -e cloud works with --no-prompt # in 1257 ms
# (in test file test/test_example_noprompt.bats, line 32)
# `solr start -e cloud --no-prompt' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr already exists.
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2 already exists.
#
# Starting up Solr on port 38713 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 38713 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 38713 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
not ok 41 Check export command # in 2955 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_export.bats, line 36)
# `assert_output --partial 'Export complete'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Export complete
# output (48 lines):
# WARN - 2026-05-26 16:33:06.026; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:38713/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:38713/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/41-1
# Last output:
# WARN - 2026-05-26 16:33:06.026; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:38713/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:38713/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
ok 42 export fails on non cloud mode # in 2769 ms
Waiting for Tika Server to be ready on port 38718
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 43 using curl to extract a single pdf file # in 16566 ms
WARNING: Docker not available; bypassing test.
ok 44 using the bin/solr post tool to extract content from pdf # in 16556 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to crawl web site # in 16565 ms
Stopping Tika Server container
not ok 46 healthcheck on cloud solr # in 1292 ms
# (in test file test/test_healthcheck.bats, line 32)
# `solr start -e films' failed
#
# Starting up Solr on port 38713 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 38713 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 38713 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/46-1
not ok 47 healthcheck errors on standalone solr # in 1366 ms
# (in test file test/test_healthcheck.bats, line 39)
# `solr start --user-managed -e films' failed
#
# Starting up Solr on port 38713 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start --user-managed -p 38713 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 38713 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/47-2
ok 48 solr help flag prints help # in 736 ms
ok 49 solr with no flags prints help # in 750 ms
ok 50 start help flag prints help # in 157 ms
ok 51 start h flag prints help # in 154 ms
ok 52 stop help flag prints help # in 144 ms
ok 53 restart help flag prints help # in 159 ms
ok 54 status help flag prints help # in 823 ms
ok 55 healthcheck help flag prints help # in 755 ms
ok 56 create help flag prints help # in 762 ms
ok 57 delete help flag prints help # in 786 ms
ok 58 zk help flag prints help # in 780 ms
ok 59 auth help flag prints help # in 754 ms
ok 60 assert help flag prints help # in 758 ms
ok 61 post help flag prints help # in 752 ms
not ok 62 SQL Module # in 18945 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 36)
# `assert_output --partial '"docs":'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# "docs":
# output (16 lines):
#
# ERROR: Error from server at http://localhost:38713/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/62-1
# WARN - 2026-05-26 16:34:14.387; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:34:14.391; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-26 16:34:30.119; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# ERROR: Error from server at http://localhost:38713/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 63 icu collation in analysis-extras module # in 17919 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 49)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/63-2
# WARN - 2026-05-26 16:34:32.338; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:34:32.342; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-26 16:34:48.071; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 64 Check lifecycle of sentiment classification # in 10320 ms
# (in test file test/test_opennlp.bats, line 63)
# `solr start -m 4g -Dsolr.modules=analysis-extras -Dsolr.packages.enabled=true' failed
# Downloading onnx model and vocab...
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 160M 0 432k 0 0 346k 0 0:07:54 0:00:01 0:07:53 346k
8 160M 8 13.2M 0 0 6155k 0 0:00:26 0:00:02 0:00:24 6155k
21 160M 21 34.3M 0 0 10.7M 0 0:00:14 0:00:03 0:00:11 10.7M
35 160M 35 56.8M 0 0 13.5M 0 0:00:11 0:00:04 0:00:07 13.5M
49 160M 49 78.9M 0 0 15.2M 0 0:00:10 0:00:05 0:00:05 15.8M
62 160M 62 100M 0 0 16.2M 0 0:00:09 0:00:06 0:00:03 20.3M
76 160M 76 123M 0 0 17.1M 0 0:00:09 0:00:07 0:00:02 22.1M
90 160M 90 145M 0 0 17.7M 0 0:00:09 0:00:08 0:00:01 22.2M
100 160M 100 160M 0 0 18.1M 0 0:00:08 0:00:08 --:--:-- 22.2M
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 851k 100 851k 0 0 665k 0 0:00:01 0:00:01 --:--:-- 665k
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/64-1
# Last output:
# total 64M
# drwxr-xr-x 2 jenkins jenkins 4.0K May 26 16:20 .
# drwxr-xr-x 3 jenkins jenkins 4.0K May 26 16:20 ..
# -rw-r--r-- 1 jenkins jenkins 14M May 26 16:20 icu4j-77.1.jar
# -rw-r--r-- 1 jenkins jenkins 83K May 26 16:20 lucene-analysis-icu-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 30K May 26 16:20 lucene-analysis-morfologik-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 40K May 26 16:20 lucene-analysis-opennlp-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 3.5M May 26 16:20 lucene-analysis-smartcn-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 508K May 26 16:20 lucene-analysis-stempel-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 19K May 26 16:20 morfologik-fsa-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 1.8M May 26 16:20 morfologik-polish-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 51K May 26 16:20 morfologik-stemming-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 4.0M May 26 16:20 morfologik-ukrainian-search-4.9.1.jar
# -rw-r--r-- 1 jenkins jenkins 39M May 26 16:20 onnxruntime-1.24.3.jar
# -rw-r--r-- 1 jenkins jenkins 28K May 26 16:20 opennlp-dl-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 1.4M May 26 16:20 opennlp-tools-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 39K May 26 16:20 solr-analysis-extras-10.1.0-SNAPSHOT.jar
not ok 65 lifecycle of package # in 2140 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_packages.bats, line 38)
# `assert_output --partial "Available packages:"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Available packages:
# output (45 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/65-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 66 deploying and undeploying a collection level package # in 1409 ms
# (in test file test/test_packages.bats, line 44)
# `solr create -c foo-1.2' failed
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38713.
#
# ERROR: Error from server at http://localhost:38713/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:38713/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/66-2
# Last output:
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
not ok 68 Affinity placement plugin using sysprop # in 21815 ms
# (in test file test/test_placement_plugin.bats, line 34)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:38713 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/68-1
# WARN - 2026-05-26 16:35:08.537; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:35:08.540; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-26 16:35:24.289; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
not ok 69 Random placement plugin using ENV # in 21717 ms
# (in test file test/test_placement_plugin.bats, line 43)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:38713 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/69-2
# WARN - 2026-05-26 16:35:30.297; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-26 16:35:30.301; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-26 16:35:46.024; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
not ok 70 setup_file failed
# (from function `setup_file' in test file test/test_post.bats, line 22)
# `solr start -Dsolr.modules=extraction' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
not ok 81 setup_file failed
# (from function `setup_file' in test file test/test_postlogs.bats, line 22)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 16 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-26T16:35:59.108416708Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-26T16:35:59.108461725Z It should be set to 65000 to avoid operational disruption.
2026-05-26T16:35:59.108464909Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-26T16:35:59.108696950Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-26T16:35:59.108702463Z
2026-05-26T16:35:59.119646151Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-26T16:35:59.127948623Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-26T16:35:59.204362519Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-26T16:35:59.236017019Z WARNING: A command line option has enabled the Security Manager
2026-05-26T16:35:59.236042263Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-26T16:36:00.467256876Z 2026-05-26 16:36:00.463 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-26T16:36:00.742234048Z 2026-05-26 16:36:00.741 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-26T16:36:00.767482907Z 2026-05-26 16:36:00.766 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-26T16:36:00.774296663Z 2026-05-26 16:36:00.773 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-26T16:36:00.774360430Z 2026-05-26 16:36:00.773 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-26T16:36:00.774479463Z 2026-05-26 16:36:00.774 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-26T16:36:00.774579072Z 2026-05-26 16:36:00.774 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-26T16:36:00.774075394Z
2026-05-26T16:36:00.777266804Z 2026-05-26 16:36:00.776 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-26T16:36:00.798303419Z 2026-05-26 16:36:00.797 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-26T16:36:00.811573451Z 2026-05-26 16:36:00.811 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-26T16:36:00.811812746Z 2026-05-26 16:36:00.811 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-26T16:36:00.846089050Z 2026-05-26 16:36:00.845 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-26T16:36:01.615148467Z 2026-05-26 16:36:01.614 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-26T16:36:01.889679899Z 2026-05-26 16:36:01.889 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-26T16:36:01.895834034Z 2026-05-26 16:36:01.895 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-26T16:36:01.895859749Z 2026-05-26 16:36:01.895 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-26T16:36:01.898854380Z 2026-05-26 16:36:01.898 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-26T16:36:01.898947925Z 2026-05-26 16:36:01.898 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-26T16:36:01.960808217Z 2026-05-26 16:36:01.960 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-26T16:36:02.401224458Z 2026-05-26 16:36:02.400 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-26T16:36:02.413777621Z 2026-05-26 16:36:02.413 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-26T16:36:02.450179751Z 2026-05-26 16:36:02.449 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-26T16:36:02.478157861Z 2026-05-26 16:36:02.477 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-26T16:36:02.478235551Z 2026-05-26 16:36:02.477 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-26T16:36:02.595381863Z 2026-05-26 16:36:02.594 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-26T16:36:02.617279742Z 2026-05-26 16:36:02.616 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-26T16:36:02.631403598Z 2026-05-26 16:36:02.630 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-26T16:36:02.642622168Z 2026-05-26 16:36:02.642 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72122242099511296-solr-node1:8983_solr-n_0000000000) starting
2026-05-26T16:36:02.712368569Z 2026-05-26 16:36:02.711 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-26T16:36:02.713215046Z 2026-05-26 16:36:02.712 INFO (OverseerStateUpdate-72122242099511296-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-26T16:36:02.723214967Z 2026-05-26 16:36:02.722 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-26T16:36:02.763798540Z 2026-05-26 16:36:02.763 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-26T16:36:03.155926434Z 2026-05-26 16:36:03.155 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-26T16:36:03.330731860Z 2026-05-26 16:36:03.330 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-26T16:36:03.618806129Z 2026-05-26 16:36:03.618 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-26T16:36:03.986022190Z 2026-05-26 16:36:03.985 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-26T16:36:04.079117588Z 2026-05-26 16:36:04.077 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_26.request.log
2026-05-26T16:36:04.082929796Z 2026-05-26 16:36:04.081 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-26T16:36:04.084546654Z 2026-05-26 16:36:04.084 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4957ms
2026-05-26T16:36:04.435678877Z 2026-05-26 16:36:04.435 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=75
2026-05-26T16:36:04.652299556Z 2026-05-26 16:36:04.651 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=25
2026-05-26T16:36:08.179673429Z 2026-05-26 16:36:08.179 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-26T16:36:08.209597741Z 2026-05-26 16:36:08.209 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-26T16:36:11.981906117Z 2026-05-26 16:36:11.980 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=13
2026-05-26T16:36:12.017549104Z 2026-05-26 16:36:12.015 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=15
2026-05-26T16:36:12.046269732Z 2026-05-26 16:36:12.045 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-26T16:36:12.293482464Z 2026-05-26 16:36:12.293 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-26T16:36:12.312941208Z 2026-05-26 16:36:12.312 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-26T16:36:12.324616382Z 2026-05-26 16:36:12.324 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-26T16:36:12.361781935Z 2026-05-26 16:36:12.361 INFO (OverseerStateUpdate-72122242099511296-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-26T16:36:12.361806608Z "core":"test-collection_shard3_replica_n1",
2026-05-26T16:36:12.361810252Z "node_name":"solr-node1:8983_solr",
2026-05-26T16:36:12.361813099Z "base_url":"http://solr-node1:8983/solr",
2026-05-26T16:36:12.361816249Z "collection":"test-collection",
2026-05-26T16:36:12.361819167Z "shard":"shard3",
2026-05-26T16:36:12.361822302Z "state":"down",
2026-05-26T16:36:12.361824959Z "type":"NRT",
2026-05-26T16:36:12.361827622Z "operation":"ADDREPLICA",
2026-05-26T16:36:12.361830528Z "waitForFinalState":"false"}
2026-05-26T16:36:12.370914902Z 2026-05-26 16:36:12.370 INFO (OverseerStateUpdate-72122242099511296-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-26T16:36:12.370932581Z "core":"test-collection_shard2_replica_n2",
2026-05-26T16:36:12.370935234Z "node_name":"solr-node3:8985_solr",
2026-05-26T16:36:12.370937891Z "base_url":"http://solr-node3:8985/solr",
2026-05-26T16:36:12.370943471Z "collection":"test-collection",
2026-05-26T16:36:12.370945500Z "shard":"shard2",
2026-05-26T16:36:12.370947571Z "state":"down",
2026-05-26T16:36:12.370949520Z "type":"NRT",
2026-05-26T16:36:12.370951490Z "operation":"ADDREPLICA",
2026-05-26T16:36:12.370953595Z "waitForFinalState":"false"}
2026-05-26T16:36:12.375700134Z 2026-05-26 16:36:12.375 INFO (OverseerStateUpdate-72122242099511296-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-26T16:36:12.375711294Z "core":"test-collection_shard1_replica_n4",
2026-05-26T16:36:12.375713476Z "node_name":"solr-node2:8984_solr",
2026-05-26T16:36:12.375726737Z "base_url":"http://solr-node2:8984/solr",
2026-05-26T16:36:12.375729033Z "collection":"test-collection",
2026-05-26T16:36:12.375730955Z "shard":"shard1",
2026-05-26T16:36:12.375732833Z "state":"down",
2026-05-26T16:36:12.375734685Z "type":"NRT",
2026-05-26T16:36:12.375736619Z "operation":"ADDREPLICA",
2026-05-26T16:36:12.375738524Z "waitForFinalState":"false"}
2026-05-26T16:36:12.484928818Z 2026-05-26 16:36:12.484 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-26T16:36:12.562870099Z 2026-05-26 16:36:12.561 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-26T16:36:12.653657612Z 2026-05-26 16:36:12.653 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-26T16:36:12.715678348Z 2026-05-26 16:36:12.715 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-26T16:36:13.191283243Z 2026-05-26 16:36:13.190 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-26T16:36:13.191308033Z 2026-05-26 16:36:13.190 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-26T16:36:13.238408689Z 2026-05-26 16:36:13.237 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-26T16:36:14.574690039Z 2026-05-26 16:36:14.573 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-26T16:36:14.610637780Z 2026-05-26 16:36:14.610 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-26T16:36:14.618320906Z 2026-05-26 16:36:14.617 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-26T16:36:14.619432354Z 2026-05-26 16:36:14.618 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-26T16:36:14.619490546Z 2026-05-26 16:36:14.619 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-26T16:36:14.721466450Z 2026-05-26 16:36:14.720 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-26T16:36:14.723835377Z 2026-05-26 16:36:14.723 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-26T16:36:14.726091402Z 2026-05-26 16:36:14.725 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-26T16:36:14.728380441Z 2026-05-26 16:36:14.727 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-26T16:36:14.747019213Z 2026-05-26 16:36:14.746 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-26T16:36:14.755154180Z 2026-05-26 16:36:14.754 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-26T16:36:14.768222658Z 2026-05-26 16:36:14.767 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-26T16:36:14.768300448Z 2026-05-26 16:36:14.767 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-26T16:36:14.778927775Z 2026-05-26 16:36:14.778 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-26T16:36:14.782054849Z 2026-05-26 16:36:14.781 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-26T16:36:14.783364912Z 2026-05-26 16:36:14.783 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-26T16:36:14.794958924Z 2026-05-26 16:36:14.794 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-26T16:36:14.794977129Z 2026-05-26 16:36:14.794 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-26T16:36:14.795074790Z 2026-05-26 16:36:14.794 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-26T16:36:14.797018252Z 2026-05-26 16:36:14.796 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-26T16:36:14.797735005Z 2026-05-26 16:36:14.797 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-26T16:36:14.799683161Z 2026-05-26 16:36:14.799 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72122242099511296-core_node3-n_0000000000
2026-05-26T16:36:14.805272068Z 2026-05-26 16:36:14.805 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-26T16:36:14.912508679Z 2026-05-26 16:36:14.912 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-26T16:36:14.916945009Z 2026-05-26 16:36:14.916 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-26T16:36:14.931529334Z 2026-05-26 16:36:14.931 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2371
2026-05-26T16:36:15.024481822Z 2026-05-26 16:36:15.023 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 95973,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-26T16:35:58.855584853Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-26T16:36:05.130774633Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-26T16:36:05.130810816Z It should be set to 65000 to avoid operational disruption.
2026-05-26T16:36:05.130815160Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-26T16:36:05.131123536Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-26T16:36:05.131141785Z
2026-05-26T16:36:05.142422834Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-26T16:36:05.150246562Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-26T16:36:05.196796574Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-26T16:36:05.234973752Z WARNING: A command line option has enabled the Security Manager
2026-05-26T16:36:05.234995070Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-26T16:36:06.480755001Z 2026-05-26 16:36:06.477 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-26T16:36:06.754197939Z 2026-05-26 16:36:06.753 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-26T16:36:06.781147122Z 2026-05-26 16:36:06.780 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-26T16:36:06.788881164Z 2026-05-26 16:36:06.788 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-26T16:36:06.788912365Z 2026-05-26 16:36:06.788 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-26T16:36:06.789118014Z 2026-05-26 16:36:06.788 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-26T16:36:06.789326158Z 2026-05-26 16:36:06.788 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-26T16:36:06.788409132Z
2026-05-26T16:36:06.792015319Z 2026-05-26 16:36:06.791 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-26T16:36:06.817245732Z 2026-05-26 16:36:06.816 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-26T16:36:06.842442944Z 2026-05-26 16:36:06.841 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-26T16:36:06.990374490Z 2026-05-26 16:36:06.989 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-26T16:36:07.012083147Z 2026-05-26 16:36:07.011 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-26T16:36:07.012192152Z 2026-05-26 16:36:07.011 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-26T16:36:07.012291509Z 2026-05-26 16:36:07.011 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-26T16:36:07.122464710Z 2026-05-26 16:36:07.121 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-26T16:36:07.122784150Z 2026-05-26 16:36:07.122 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-26T16:36:07.122869483Z 2026-05-26 16:36:07.122 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-26T16:36:07.157683951Z 2026-05-26 16:36:07.157 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-26T16:36:07.922560110Z 2026-05-26 16:36:07.921 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-26T16:36:08.070692318Z 2026-05-26 16:36:08.070 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-26T16:36:08.072231567Z 2026-05-26 16:36:08.070 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-26T16:36:08.082989613Z 2026-05-26 16:36:08.082 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-26T16:36:08.089322061Z 2026-05-26 16:36:08.088 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-26T16:36:08.093657087Z 2026-05-26 16:36:08.093 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-26T16:36:08.093790520Z 2026-05-26 16:36:08.093 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-26T16:36:08.132742492Z 2026-05-26 16:36:08.132 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-26T16:36:08.143491124Z 2026-05-26 16:36:08.142 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-26T16:36:08.173350225Z 2026-05-26 16:36:08.172 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-26T16:36:08.180515663Z 2026-05-26 16:36:08.180 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-26T16:36:08.188159780Z 2026-05-26 16:36:08.187 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-26T16:36:08.199376444Z 2026-05-26 16:36:08.198 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-26T16:36:08.209721082Z 2026-05-26 16:36:08.209 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-26T16:36:08.305630000Z 2026-05-26 16:36:08.305 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-26T16:36:08.663429829Z 2026-05-26 16:36:08.662 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-26T16:36:08.840761705Z 2026-05-26 16:36:08.840 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-26T16:36:09.117896544Z 2026-05-26 16:36:09.117 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-26T16:36:09.249661265Z 2026-05-26 16:36:09.248 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-26T16:36:09.255373762Z 2026-05-26 16:36:09.255 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_26.request.log
2026-05-26T16:36:09.259008525Z 2026-05-26 16:36:09.258 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-26T16:36:09.260510510Z 2026-05-26 16:36:09.260 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4111ms
2026-05-26T16:36:09.425552886Z 2026-05-26 16:36:09.425 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=47
2026-05-26T16:36:09.479386394Z 2026-05-26 16:36:09.478 INFO (qtp985324122-56-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-05-26T16:36:12.563439929Z 2026-05-26 16:36:12.562 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-26T16:36:12.612384422Z 2026-05-26 16:36:12.611 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-26T16:36:12.670320079Z 2026-05-26 16:36:12.669 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-26T16:36:13.104769969Z 2026-05-26 16:36:13.104 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-26T16:36:13.104853109Z 2026-05-26 16:36:13.104 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-26T16:36:13.124093534Z 2026-05-26 16:36:13.123 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-26T16:36:13.567599795Z 2026-05-26 16:36:13.567 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-26T16:36:13.586827941Z 2026-05-26 16:36:13.586 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-26T16:36:13.590721355Z 2026-05-26 16:36:13.590 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-26T16:36:13.591866710Z 2026-05-26 16:36:13.591 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-26T16:36:13.591896313Z 2026-05-26 16:36:13.591 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-26T16:36:13.638657479Z 2026-05-26 16:36:13.638 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-26T16:36:13.642365788Z 2026-05-26 16:36:13.642 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-26T16:36:13.645275898Z 2026-05-26 16:36:13.644 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-26T16:36:13.648048398Z 2026-05-26 16:36:13.647 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-26T16:36:13.661166076Z 2026-05-26 16:36:13.660 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-26T16:36:13.669386964Z 2026-05-26 16:36:13.668 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-26T16:36:13.676294774Z 2026-05-26 16:36:13.675 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-26T16:36:13.676313939Z 2026-05-26 16:36:13.675 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-26T16:36:13.685885086Z 2026-05-26 16:36:13.685 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Register
[...truncated too long message...]
ntry
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/93-7
# Last output:
# INFO - 2026-05-26 16:37:19.435; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-05-26 16:37:19.438; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-05-26 16:37:19.450; org.apache.solr.common.util.EnvUtils; You are passing in deprecated system property host and should upgrade to using solr.host.advertise instead. The deprecated property support will be removed in future version of Solr.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is https://localhost:38713.
#
# ERROR: IOException occurred when talking to server at: https://localhost:38713/solr/admin/info/system
not ok 94 SOLR-11740 check 'solr stop' connection # in 231 ms
# (in test file test/test_start_solr.bats, line 32)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/94-1
not ok 95 stop command for single port # in 230 ms
# (in test file test/test_start_solr.bats, line 45)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/95-2
ok 96 check stop command doesn't hang # in 218 ms
not ok 97 SOLR-16976 solr starts with remote JMX enabled # in 228 ms
# (in test file test/test_start_solr.bats, line 77)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/97-4
not ok 98 deprecated system properties converted to modern properties # in 233 ms
# (in test file test/test_start_solr.bats, line 85)
# `solr start -Ddisable.config.edit=true' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/98-5
not ok 99 start with custom jetty options # in 236 ms
# (in test file test/test_start_solr.bats, line 93)
# `solr start --jettyconfig "--module=server"' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/99-6
not ok 100 bootstrapping a configset # in 234 ms
# (in test file test/test_start_solr.bats, line 104)
# `solr start -Dbootstrap_confdir="${confdir_path}" -Dcollection.configName=techproducts' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/100-7
not ok 101 status detects locally running solr # in 1018 ms
# (in test file test/test_status.bats, line 34)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/101-1
# Last output:
#
# No Solr nodes are running.
not ok 102 status with --solr-url from user # in 226 ms
# (in test file test/test_status.bats, line 43)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/102-2
not ok 103 status with --port from user # in 232 ms
# (in test file test/test_status.bats, line 50)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-3
ok 104 multiple connection options are prevented # in 881 ms
ok 105 status with invalid --solr-url from user # in 1279 ms
not ok 106 status with --short format # in 230 ms
# (in test file test/test_status.bats, line 67)
# `solr start' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-6
not ok 107 setup_file failed
# (from function `setup_file' in test file test/test_stream.bats, line 22)
# `solr start -e techproducts' failed
#
# Starting up Solr on port 38713 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 38713 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" -Dsolr.modules=clustering,extraction,langid,ltr,scripting -Dsolr.ltr.enabled=true -Dsolr.clustering.enabled=true
#
#
# ERROR: Port 38713 is already being used by another process.
#
ok 111 --version returns Solr version # in 749 ms
ok 112 version as direct tool call still runs # in 746 ms
not ok 113 setup_file failed
# (from function `setup_file' in test file test/test_zk.bats, line 22)
# `solr start -DminStateByteLenForCompression=0' failed
#
# Port 38713 is already being used by another process (pid: 87460)
# Please choose a different port using the -p option.
#
ok 125 Cleanup # in 1872 ms
FATAL: command execution failed
java.io.IOException: Backing channel 'lucene-solr-4' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:227)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:306)
at jdk.proxy3/jdk.proxy3.$Proxy270.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1212)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1204)
at hudson.Launcher$ProcStarter.join(Launcher.java:521)
at PluginClassLoader for gradle//hudson.plugins.gradle.Gradle.perform(Gradle.java:321)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:164)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:527)
at hudson.model.Run.execute(Run.java:1833)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:446)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:80)
Caused by: java.io.EOFException
at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2915)
at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3410)
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:954)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:392)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50)
at hudson.remoting.Command.readFrom(Command.java:141)
at hudson.remoting.Command.readFrom(Command.java:127)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:62)
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
ERROR: Step ‘Archive the artifacts’ failed: no workspace for Solr/Solr-TestIntegration-10.x #217
ERROR: lucene-solr-4 is offline; cannot locate jdk_21_latest
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
ERROR: lucene-solr-4 is offline; cannot locate jdk_21_latest
ERROR: lucene-solr-4 is offline; cannot locate jdk_21_latest
ERROR: lucene-solr-4 is offline; cannot locate jdk_21_latest
ERROR: lucene-solr-4 is offline; cannot locate jdk_21_latest
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 216 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/216/
No tests ran.
Build Log:
[...truncated 373 lines...]
# ERROR: Solr is not running on url http://localhost:46437 after 5 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/4-2
# Last output:
# Waiting up to 180 seconds to see Solr running on port 46437 [|] [/] [-] [\] [|]
# Started Solr server on port 46437 (pid=2012562). Happy searching!
#
#
ok 5 auth rejects block-unknown option with invalid boolean # in 1798 ms
ok 6 auth rejects update-include-file-only option with invalid boolean # in 1770 ms
not ok 7 auth enable/disable lifecycle # in 20605 ms
# (in test file test/test_auth.bats, line 40)
# `solr auth enable --type basicAuth --credentials name:password' failed
# Waiting up to 180 seconds to see Solr running on port 46437 [|] [/] [-] [\] [|]
# Started Solr server on port 46437 (pid=2014834). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
# Couldn't get ZooKeeper host. Please make sure Solr is running in cloud mode, or a zk-host has been passed in.
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Server refused connection at: http://localhost:46437/solr/admin/info/system
not ok 8 enable auth connects via zookeeper # in 1196 ms
# (in test file test/test_auth.bats, line 53)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 9 create, config, api, and delete with basic auth # in 16678 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-22 16:50:50.552; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:50:50.556; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 10 post, postlogs and export with basic auth # in 16638 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/10-2
# Last output:
# WARN - 2026-05-22 16:51:07.238; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:51:07.241; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 11 package with basic auth # in 16665 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/11-3
# Last output:
# WARN - 2026-05-22 16:51:23.925; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:51:23.928; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 12 admin UI static content is excepted from basic auth # in 16671 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/12-4
# Last output:
# WARN - 2026-05-22 16:51:40.596; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:51:40.599; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 13 setup_file failed
# (from function `setup_file' in test file test/test_bats.bats, line 26)
# `solr start --verbose' failed
# Using Solr root directory: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT
# Using Java: /home/jenkins/tools/java/latest21/bin/java
# openjdk version "21.0.4" 2024-07-16 LTS
# OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
# OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
not ok 14 setup_file failed
# (from function `setup_file' in test file test/test_compression.bats, line 22)
# `solr start -e films' failed
#
# Starting up Solr on port 46437 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46437 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46437 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test # at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/-
not ok 17 setup_file failed
# (from function `setup_file' in test file test/test_config.bats, line 22)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
not ok 20 create for non cloud mode # in 1518 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 34)
# `assert_output --partial "Created new core 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created new core 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/20-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 21 create for cloud mode # in 1451 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 40)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/21-2
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
ok 22 multiple connection options are prevented # in 847 ms
not ok 23 setup_file failed
# (from function `setup_file' in test file test/test_create_collection.bats, line 22)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
not ok 34 setup_file failed
# (from function `setup_file' in test file test/test_delete_collection.bats, line 22)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
not ok 39 start -e cloud works with --jvm-opts # in 1275 ms
# (in test file test/test_example.bats, line 32)
# `solr start -e cloud --no-prompt --jvm-opts "-Dcustom.prop=helloworld"' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Creating Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr
# Cloning /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1 into
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2
#
# Starting up Solr on port 46437 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46437 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" --jvm-opts "-Dcustom.prop=helloworld"
#
#
# ERROR: Port 46437 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/39-1
not ok 40 start -e cloud works with --no-prompt # in 1247 ms
# (in test file test/test_example_noprompt.bats, line 32)
# `solr start -e cloud --no-prompt' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr already exists.
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2 already exists.
#
# Starting up Solr on port 46437 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46437 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46437 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
not ok 41 Check export command # in 3002 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_export.bats, line 36)
# `assert_output --partial 'Export complete'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Export complete
# output (48 lines):
# WARN - 2026-05-22 16:52:10.986; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:46437/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46437/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/41-1
# Last output:
# WARN - 2026-05-22 16:52:10.986; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:46437/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46437/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
ok 42 export fails on non cloud mode # in 2865 ms
Waiting for Tika Server to be ready on port 46442
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 43 using curl to extract a single pdf file # in 16594 ms
WARNING: Docker not available; bypassing test.
ok 44 using the bin/solr post tool to extract content from pdf # in 16549 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to crawl web site # in 16537 ms
Stopping Tika Server container
not ok 46 healthcheck on cloud solr # in 1259 ms
# (in test file test/test_healthcheck.bats, line 32)
# `solr start -e films' failed
#
# Starting up Solr on port 46437 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46437 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46437 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/46-1
not ok 47 healthcheck errors on standalone solr # in 1255 ms
# (in test file test/test_healthcheck.bats, line 39)
# `solr start --user-managed -e films' failed
#
# Starting up Solr on port 46437 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start --user-managed -p 46437 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46437 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/47-2
ok 48 solr help flag prints help # in 740 ms
ok 49 solr with no flags prints help # in 759 ms
ok 50 start help flag prints help # in 147 ms
ok 51 start h flag prints help # in 149 ms
ok 52 stop help flag prints help # in 144 ms
ok 53 restart help flag prints help # in 158 ms
ok 54 status help flag prints help # in 805 ms
ok 55 healthcheck help flag prints help # in 749 ms
ok 56 create help flag prints help # in 759 ms
ok 57 delete help flag prints help # in 789 ms
ok 58 zk help flag prints help # in 788 ms
ok 59 auth help flag prints help # in 761 ms
ok 60 assert help flag prints help # in 740 ms
ok 61 post help flag prints help # in 749 ms
not ok 62 SQL Module # in 18885 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 36)
# `assert_output --partial '"docs":'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# "docs":
# output (16 lines):
#
# ERROR: Error from server at http://localhost:46437/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/62-1
# WARN - 2026-05-22 16:53:19.275; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:53:19.278; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-22 16:53:35.006; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# ERROR: Error from server at http://localhost:46437/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 63 icu collation in analysis-extras module # in 17864 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 49)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/63-2
# WARN - 2026-05-22 16:53:37.170; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:53:37.173; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-22 16:53:52.896; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 64 Check lifecycle of sentiment classification # in 10777 ms
# (in test file test/test_opennlp.bats, line 63)
# `solr start -m 4g -Dsolr.modules=analysis-extras -Dsolr.packages.enabled=true' failed
# Downloading onnx model and vocab...
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 160M 0 208k 0 0 180k 0 0:15:09 0:00:01 0:15:08 180k
7 160M 7 12.6M 0 0 6077k 0 0:00:27 0:00:02 0:00:25 6076k
20 160M 20 33.2M 0 0 10.2M 0 0:00:15 0:00:03 0:00:12 10.2M
34 160M 34 54.7M 0 0 12.8M 0 0:00:12 0:00:04 0:00:08 12.8M
47 160M 47 75.8M 0 0 14.4M 0 0:00:11 0:00:05 0:00:06 15.2M
60 160M 60 96.5M 0 0 15.5M 0 0:00:10 0:00:06 0:00:04 19.0M
72 160M 72 117M 0 0 16.2M 0 0:00:09 0:00:07 0:00:02 20.6M
85 160M 85 137M 0 0 16.8M 0 0:00:09 0:00:08 0:00:01 21.2M
97 160M 97 156M 0 0 17.0M 0 0:00:09 0:00:09 --:--:-- 20.7M
100 160M 100 160M 0 0 17.2M 0 0:00:09 0:00:09 --:--:-- 20.8M
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11 851k 11 98304 0 0 102k 0 0:00:08 --:--:-- 0:00:08 102k
100 851k 100 851k 0 0 668k 0 0:00:01 0:00:01 --:--:-- 668k
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/64-1
# Last output:
# total 64M
# drwxr-xr-x 2 jenkins jenkins 4.0K May 22 16:39 .
# drwxr-xr-x 3 jenkins jenkins 4.0K May 22 16:39 ..
# -rw-r--r-- 1 jenkins jenkins 14M May 22 16:39 icu4j-77.1.jar
# -rw-r--r-- 1 jenkins jenkins 83K May 22 16:39 lucene-analysis-icu-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 30K May 22 16:39 lucene-analysis-morfologik-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 40K May 22 16:39 lucene-analysis-opennlp-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 3.5M May 22 16:39 lucene-analysis-smartcn-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 508K May 22 16:39 lucene-analysis-stempel-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 19K May 22 16:39 morfologik-fsa-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 1.8M May 22 16:39 morfologik-polish-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 51K May 22 16:39 morfologik-stemming-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 4.0M May 22 16:39 morfologik-ukrainian-search-4.9.1.jar
# -rw-r--r-- 1 jenkins jenkins 39M May 22 16:39 onnxruntime-1.24.3.jar
# -rw-r--r-- 1 jenkins jenkins 28K May 22 16:39 opennlp-dl-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 1.4M May 22 16:39 opennlp-tools-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 39K May 22 16:39 solr-analysis-extras-10.1.0-SNAPSHOT.jar
not ok 65 lifecycle of package # in 2283 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_packages.bats, line 38)
# `assert_output --partial "Available packages:"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Available packages:
# output (45 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/65-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 66 deploying and undeploying a collection level package # in 1295 ms
# (in test file test/test_packages.bats, line 44)
# `solr create -c foo-1.2' failed
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46437.
#
# ERROR: Error from server at http://localhost:46437/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46437/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/66-2
# Last output:
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
not ok 68 Affinity placement plugin using sysprop # in 21796 ms
# (in test file test/test_placement_plugin.bats, line 34)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:46437 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/68-1
# WARN - 2026-05-22 16:54:13.843; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:54:13.846; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-22 16:54:29.580; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
not ok 69 Random placement plugin using ENV # in 21791 ms
# (in test file test/test_placement_plugin.bats, line 43)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:46437 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/69-2
# WARN - 2026-05-22 16:54:35.669; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-22 16:54:35.672; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-22 16:54:51.394; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
not ok 70 setup_file failed
# (from function `setup_file' in test file test/test_post.bats, line 22)
# `solr start -Dsolr.modules=extraction' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
not ok 81 setup_file failed
# (from function `setup_file' in test file test/test_postlogs.bats, line 22)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-22T16:54:59.614966733Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-22T16:54:59.615010017Z It should be set to 65000 to avoid operational disruption.
2026-05-22T16:54:59.615014615Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-22T16:54:59.615342237Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-22T16:54:59.615351837Z
2026-05-22T16:54:59.626437644Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-22T16:54:59.634790203Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-22T16:54:59.696675922Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-22T16:54:59.733483719Z WARNING: A command line option has enabled the Security Manager
2026-05-22T16:54:59.733507280Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-22T16:55:00.967462521Z 2026-05-22 16:55:00.963 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-22T16:55:01.250619921Z 2026-05-22 16:55:01.250 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-22T16:55:01.277185495Z 2026-05-22 16:55:01.276 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-22T16:55:01.285473517Z 2026-05-22 16:55:01.284 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-22T16:55:01.285586361Z 2026-05-22 16:55:01.284 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-22T16:55:01.285856282Z 2026-05-22 16:55:01.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-22T16:55:01.285999840Z 2026-05-22 16:55:01.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-22T16:55:01.285191419Z
2026-05-22T16:55:01.289402011Z 2026-05-22 16:55:01.288 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-22T16:55:01.312479852Z 2026-05-22 16:55:01.311 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-22T16:55:01.326752706Z 2026-05-22 16:55:01.326 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-22T16:55:01.326831042Z 2026-05-22 16:55:01.326 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-22T16:55:01.363745242Z 2026-05-22 16:55:01.362 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-22T16:55:02.383646292Z 2026-05-22 16:55:02.382 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-22T16:55:02.651634950Z 2026-05-22 16:55:02.651 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-22T16:55:02.656666509Z 2026-05-22 16:55:02.656 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-22T16:55:02.656737271Z 2026-05-22 16:55:02.656 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-22T16:55:02.659398754Z 2026-05-22 16:55:02.659 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-22T16:55:02.659467657Z 2026-05-22 16:55:02.659 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-22T16:55:02.706256296Z 2026-05-22 16:55:02.705 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-22T16:55:03.161900667Z 2026-05-22 16:55:03.161 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-22T16:55:03.174895522Z 2026-05-22 16:55:03.174 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-22T16:55:03.210964018Z 2026-05-22 16:55:03.210 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-22T16:55:03.239099653Z 2026-05-22 16:55:03.238 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-22T16:55:03.239165246Z 2026-05-22 16:55:03.238 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-22T16:55:03.358411922Z 2026-05-22 16:55:03.357 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-22T16:55:03.379202699Z 2026-05-22 16:55:03.378 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-22T16:55:03.390473736Z 2026-05-22 16:55:03.390 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-22T16:55:03.397092876Z 2026-05-22 16:55:03.396 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72099667617579008-solr-node1:8983_solr-n_0000000000) starting
2026-05-22T16:55:03.456309100Z 2026-05-22 16:55:03.455 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-22T16:55:03.457173535Z 2026-05-22 16:55:03.456 INFO (OverseerStateUpdate-72099667617579008-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-22T16:55:03.466069958Z 2026-05-22 16:55:03.465 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-22T16:55:03.506117388Z 2026-05-22 16:55:03.505 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-22T16:55:03.923644813Z 2026-05-22 16:55:03.923 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-22T16:55:04.117432096Z 2026-05-22 16:55:04.116 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-22T16:55:04.378630848Z 2026-05-22 16:55:04.378 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-22T16:55:04.504977092Z 2026-05-22 16:55:04.504 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-22T16:55:04.510640076Z 2026-05-22 16:55:04.510 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_22.request.log
2026-05-22T16:55:04.514281820Z 2026-05-22 16:55:04.513 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-22T16:55:04.515307419Z 2026-05-22 16:55:04.515 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4882ms
2026-05-22T16:55:04.967627018Z 2026-05-22 16:55:04.966 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=57
2026-05-22T16:55:05.303547575Z 2026-05-22 16:55:05.302 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-05-22T16:55:08.702827449Z 2026-05-22 16:55:08.702 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-22T16:55:08.975431597Z 2026-05-22 16:55:08.974 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-22T16:55:12.615820514Z 2026-05-22 16:55:12.614 INFO (qtp985324122-100-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=13
2026-05-22T16:55:12.723192990Z 2026-05-22 16:55:12.722 INFO (qtp985324122-100-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=79
2026-05-22T16:55:12.750936067Z 2026-05-22 16:55:12.750 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-22T16:55:13.415533728Z 2026-05-22 16:55:13.414 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-22T16:55:13.488192619Z 2026-05-22 16:55:13.487 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-22T16:55:13.506750307Z 2026-05-22 16:55:13.505 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-22T16:55:13.584812755Z 2026-05-22 16:55:13.583 INFO (OverseerStateUpdate-72099667617579008-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-22T16:55:13.584834360Z "core":"test-collection_shard3_replica_n1",
2026-05-22T16:55:13.584838783Z "node_name":"solr-node1:8983_solr",
2026-05-22T16:55:13.584842205Z "base_url":"http://solr-node1:8983/solr",
2026-05-22T16:55:13.584849567Z "collection":"test-collection",
2026-05-22T16:55:13.584852930Z "shard":"shard3",
2026-05-22T16:55:13.584856001Z "state":"down",
2026-05-22T16:55:13.584859127Z "type":"NRT",
2026-05-22T16:55:13.584862424Z "operation":"ADDREPLICA",
2026-05-22T16:55:13.584865617Z "waitForFinalState":"false"}
2026-05-22T16:55:13.615254983Z 2026-05-22 16:55:13.614 INFO (OverseerStateUpdate-72099667617579008-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-22T16:55:13.615273997Z "core":"test-collection_shard2_replica_n2",
2026-05-22T16:55:13.615276587Z "node_name":"solr-node3:8985_solr",
2026-05-22T16:55:13.615278790Z "base_url":"http://solr-node3:8985/solr",
2026-05-22T16:55:13.615281122Z "collection":"test-collection",
2026-05-22T16:55:13.615283238Z "shard":"shard2",
2026-05-22T16:55:13.615285262Z "state":"down",
2026-05-22T16:55:13.615287290Z "type":"NRT",
2026-05-22T16:55:13.615293007Z "operation":"ADDREPLICA",
2026-05-22T16:55:13.615295057Z "waitForFinalState":"false"}
2026-05-22T16:55:13.621389771Z 2026-05-22 16:55:13.620 INFO (OverseerStateUpdate-72099667617579008-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-22T16:55:13.621406547Z "core":"test-collection_shard1_replica_n4",
2026-05-22T16:55:13.621409293Z "node_name":"solr-node2:8984_solr",
2026-05-22T16:55:13.621425234Z "base_url":"http://solr-node2:8984/solr",
2026-05-22T16:55:13.621427528Z "collection":"test-collection",
2026-05-22T16:55:13.621429876Z "shard":"shard1",
2026-05-22T16:55:13.621431947Z "state":"down",
2026-05-22T16:55:13.621433858Z "type":"NRT",
2026-05-22T16:55:13.621435921Z "operation":"ADDREPLICA",
2026-05-22T16:55:13.621437869Z "waitForFinalState":"false"}
2026-05-22T16:55:13.733700666Z 2026-05-22 16:55:13.733 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-22T16:55:13.861071771Z 2026-05-22 16:55:13.860 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-22T16:55:13.931719158Z 2026-05-22 16:55:13.930 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-22T16:55:14.005552969Z 2026-05-22 16:55:14.004 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-22T16:55:14.497900169Z 2026-05-22 16:55:14.497 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-22T16:55:14.498362018Z 2026-05-22 16:55:14.497 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-22T16:55:14.522895142Z 2026-05-22 16:55:14.522 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-22T16:55:15.601362391Z 2026-05-22 16:55:15.600 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-22T16:55:15.661214223Z 2026-05-22 16:55:15.660 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-22T16:55:15.672291346Z 2026-05-22 16:55:15.671 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-22T16:55:15.674044840Z 2026-05-22 16:55:15.673 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-22T16:55:15.674058205Z 2026-05-22 16:55:15.673 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-22T16:55:15.800338957Z 2026-05-22 16:55:15.799 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-22T16:55:15.806330026Z 2026-05-22 16:55:15.805 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-22T16:55:15.809085002Z 2026-05-22 16:55:15.808 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-22T16:55:15.822935144Z 2026-05-22 16:55:15.822 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-22T16:55:15.841950536Z 2026-05-22 16:55:15.841 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-22T16:55:15.855698638Z 2026-05-22 16:55:15.855 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-22T16:55:15.863548728Z 2026-05-22 16:55:15.863 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-22T16:55:15.863571766Z 2026-05-22 16:55:15.863 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-22T16:55:15.884887055Z 2026-05-22 16:55:15.884 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-22T16:55:15.885992589Z 2026-05-22 16:55:15.885 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-22T16:55:15.886257038Z 2026-05-22 16:55:15.885 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-22T16:55:15.915975996Z 2026-05-22 16:55:15.915 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-22T16:55:15.915996061Z 2026-05-22 16:55:15.915 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-22T16:55:15.916000723Z 2026-05-22 16:55:15.915 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-22T16:55:15.917245991Z 2026-05-22 16:55:15.916 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-22T16:55:15.918251588Z 2026-05-22 16:55:15.918 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-22T16:55:15.919333739Z 2026-05-22 16:55:15.919 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72099667617579008-core_node3-n_0000000000
2026-05-22T16:55:15.927833068Z 2026-05-22 16:55:15.927 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-22T16:55:16.032416987Z 2026-05-22 16:55:16.031 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 240] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-22T16:55:16.036422778Z 2026-05-22 16:55:16.036 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-22T16:55:16.043241969Z 2026-05-22 16:55:16.042 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2184
2026-05-22T16:55:16.144673223Z 2026-05-22 16:55:16.144 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 246] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2023020,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-22T16:54:59.326387951Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-22T16:55:05.640388142Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-22T16:55:05.640412124Z It should be set to 65000 to avoid operational disruption.
2026-05-22T16:55:05.640415059Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-22T16:55:05.640752986Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-22T16:55:05.640765249Z
2026-05-22T16:55:05.650549457Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-22T16:55:05.658139379Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-22T16:55:05.729484755Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-22T16:55:05.761550942Z WARNING: A command line option has enabled the Security Manager
2026-05-22T16:55:05.761588322Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-22T16:55:06.903049235Z 2026-05-22 16:55:06.897 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-22T16:55:07.197422025Z 2026-05-22 16:55:07.196 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-22T16:55:07.223287773Z 2026-05-22 16:55:07.222 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-22T16:55:07.230399017Z 2026-05-22 16:55:07.229 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-22T16:55:07.230536022Z 2026-05-22 16:55:07.230 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-22T16:55:07.230553396Z 2026-05-22 16:55:07.230 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-22T16:55:07.230708812Z 2026-05-22 16:55:07.230 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-22T16:55:07.230221595Z
2026-05-22T16:55:07.233667076Z 2026-05-22 16:55:07.233 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-22T16:55:07.255405062Z 2026-05-22 16:55:07.254 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-22T16:55:07.278610936Z 2026-05-22 16:55:07.278 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-22T16:55:07.420673164Z 2026-05-22 16:55:07.420 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-22T16:55:07.444819619Z 2026-05-22 16:55:07.444 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-22T16:55:07.444951474Z 2026-05-22 16:55:07.444 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-22T16:55:07.445113061Z 2026-05-22 16:55:07.444 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-22T16:55:07.557249553Z 2026-05-22 16:55:07.556 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-22T16:55:07.557484121Z 2026-05-22 16:55:07.557 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-22T16:55:07.557707282Z 2026-05-22 16:55:07.557 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-22T16:55:07.603653847Z 2026-05-22 16:55:07.603 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-22T16:55:08.420722243Z 2026-05-22 16:55:08.420 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-22T16:55:08.579854907Z 2026-05-22 16:55:08.579 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-22T16:55:08.581630968Z 2026-05-22 16:55:08.579 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-22T16:55:08.590115186Z 2026-05-22 16:55:08.589 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-22T16:55:08.596378229Z 2026-05-22 16:55:08.595 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-22T16:55:08.599945885Z 2026-05-22 16:55:08.599 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-22T16:55:08.600017169Z 2026-05-22 16:55:08.599 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-22T16:55:08.636259502Z 2026-05-22 16:55:08.635 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-22T16:55:08.645052577Z 2026-05-22 16:55:08.644 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-22T16:55:08.667645930Z 2026-05-22 16:55:08.667 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-22T16:55:08.680694602Z 2026-05-22 16:55:08.680 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-22T16:55:08.691767097Z 2026-05-22 16:55:08.691 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-22T16:55:08.704577854Z 2026-05-22 16:55:08.704 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-22T16:55:08.756862195Z 2026-05-22 16:55:08.756 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-22T16:55:08.975459244Z 2026-05-22 16:55:08.974 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-22T16:55:09.069824907Z 2026-05-22 16:55:09.069 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-22T16:55:09.261696392Z 2026-05-22 16:55:09.261 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-22T16:55:09.543595807Z 2026-05-22 16:55:09.543 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-22T16:55:09.680418921Z 2026-05-22 16:55:09.679 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-22T16:55:09.686344039Z 2026-05-22 16:55:09.685 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_22.request.log
2026-05-22T16:55:09.690409937Z 2026-05-22 16:55:09.689 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-22T16:55:09.691711506Z 2026-05-22 16:55:09.691 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4034ms
2026-05-22T16:55:09.860620500Z 2026-05-22 16:55:09.859 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=47
2026-05-22T16:55:09.919198214Z 2026-05-22 16:55:09.918 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-05-22T16:55:13.859629524Z 2026-05-22 16:55:13.858 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-22T16:55:13.921664660Z 2026-05-22 16:55:13.921 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-22T16:55:13.975853361Z 2026-05-22 16:55:13.975 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-22T16:55:14.298678219Z 2026-05-22 16:55:14.297 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-22T16:55:14.298702285Z 2026-05-22 16:55:14.298 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-22T16:55:14.317726570Z 2026-05-22 16:55:14.317 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-22T16:55:14.774255171Z 2026-05-22 16:55:14.773 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-22T16:55:14.792786027Z 2026-05-22 16:55:14.792 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-22T16:55:14.796243010Z 2026-05-22 16:55:14.795 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-22T16:55:14.797294755Z 2026-05-22 16:55:14.796 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-22T16:55:14.797348882Z 2026-05-22 16:55:14.797 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-22T16:55:14.839631279Z 2026-05-22 16:55:14.839 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-22T16:55:14.844353084Z 2026-05-22 16:55:14.843 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-22T16:55:14.846470765Z 2026-05-22 16:55:14.846 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-22T16:55:14.848404456Z 2026-05-22 16:55:14.848 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-22T16:55:14.860604905Z 2026-05-22 16:55:14.859 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-22T16:55:14.869185386Z 2026-05-22 16:55:14.868 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-22T16:55:14.875449687Z 2026-05-22 16:55:14.874 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-22T16:55:14.875525058Z 2026-05-22 16:55:14.875 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-22T16:55:14.882215720Z 2026-05-22 16:55:14.881 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_so
[...truncated too long message...]
r and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Certificate was added to keystore
# Certificate was added to keystore
# Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Certificate was added to keystore
# Certificate was added to keystore
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/92-6
# Last output:
# INFO - 2026-05-22 16:56:17.909; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-05-22 16:56:17.912; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-05-22 16:56:17.924; org.apache.solr.common.util.EnvUtils; You are passing in deprecated system property host and should upgrade to using solr.host.advertise instead. The deprecated property support will be removed in future version of Solr.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is https://localhost:46437.
#
# ERROR: IOException occurred when talking to server at: https://localhost:46437/solr/admin/info/system
not ok 93 test keystore reload # in 2772 ms
# (in test file test/test_ssl.bats, line 529)
# `solr start --jvm-opts "-Dsolr.jetty.ssl.context.reload.scan.interval.secs=1 -DsocketTimeout=5000"' failed
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/93-7
# Last output:
# INFO - 2026-05-22 16:56:20.738; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-05-22 16:56:20.741; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-05-22 16:56:20.753; org.apache.solr.common.util.EnvUtils; You are passing in deprecated system property host and should upgrade to using solr.host.advertise instead. The deprecated property support will be removed in future version of Solr.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is https://localhost:46437.
#
# ERROR: IOException occurred when talking to server at: https://localhost:46437/solr/admin/info/system
not ok 94 SOLR-11740 check 'solr stop' connection # in 232 ms
# (in test file test/test_start_solr.bats, line 32)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/94-1
not ok 95 stop command for single port # in 238 ms
# (in test file test/test_start_solr.bats, line 45)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/95-2
ok 96 check stop command doesn't hang # in 239 ms
not ok 97 SOLR-16976 solr starts with remote JMX enabled # in 236 ms
# (in test file test/test_start_solr.bats, line 77)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/97-4
not ok 98 deprecated system properties converted to modern properties # in 239 ms
# (in test file test/test_start_solr.bats, line 85)
# `solr start -Ddisable.config.edit=true' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/98-5
not ok 99 start with custom jetty options # in 236 ms
# (in test file test/test_start_solr.bats, line 93)
# `solr start --jettyconfig "--module=server"' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/99-6
not ok 100 bootstrapping a configset # in 227 ms
# (in test file test/test_start_solr.bats, line 104)
# `solr start -Dbootstrap_confdir="${confdir_path}" -Dcollection.configName=techproducts' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/100-7
not ok 101 status detects locally running solr # in 980 ms
# (in test file test/test_status.bats, line 34)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/101-1
# Last output:
#
# No Solr nodes are running.
not ok 102 status with --solr-url from user # in 223 ms
# (in test file test/test_status.bats, line 43)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/102-2
not ok 103 status with --port from user # in 228 ms
# (in test file test/test_status.bats, line 50)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-3
ok 104 multiple connection options are prevented # in 848 ms
ok 105 status with invalid --solr-url from user # in 1240 ms
not ok 106 status with --short format # in 225 ms
# (in test file test/test_status.bats, line 67)
# `solr start' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-6
not ok 107 setup_file failed
# (from function `setup_file' in test file test/test_stream.bats, line 22)
# `solr start -e techproducts' failed
#
# Starting up Solr on port 46437 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46437 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" -Dsolr.modules=clustering,extraction,langid,ltr,scripting -Dsolr.ltr.enabled=true -Dsolr.clustering.enabled=true
#
#
# ERROR: Port 46437 is already being used by another process.
#
ok 111 --version returns Solr version # in 773 ms
ok 112 version as direct tool call still runs # in 777 ms
not ok 113 setup_file failed
# (from function `setup_file' in test file test/test_zk.bats, line 22)
# `solr start -DminStateByteLenForCompression=0' failed
#
# Port 46437 is already being used by another process (pid: 2014834)
# Please choose a different port using the -p option.
#
ok 125 Cleanup # in 1871 ms
Build timed out (after 5,175 minutes). Marking the build as aborted.
Build timed out (after 5,175 minutes). Marking the build as failed.
Build was aborted
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 215 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/215/
No tests ran.
Build Log:
[...truncated 372 lines...]
# ERROR: Solr is not running on url http://localhost:46519 after 5 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/4-2
# Last output:
# Waiting up to 180 seconds to see Solr running on port 46519 [|] [/] [-] [\] [|]
# Started Solr server on port 46519 (pid=1319852). Happy searching!
#
#
ok 5 auth rejects block-unknown option with invalid boolean # in 1836 ms
ok 6 auth rejects update-include-file-only option with invalid boolean # in 1831 ms
not ok 7 auth enable/disable lifecycle # in 20823 ms
# (in test file test/test_auth.bats, line 40)
# `solr auth enable --type basicAuth --credentials name:password' failed
# Waiting up to 180 seconds to see Solr running on port 46519 [|] [/] [-] [\] [|]
# Started Solr server on port 46519 (pid=1324609). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
# Couldn't get ZooKeeper host. Please make sure Solr is running in cloud mode, or a zk-host has been passed in.
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Server refused connection at: http://localhost:46519/solr/admin/info/system
not ok 8 enable auth connects via zookeeper # in 1294 ms
# (in test file test/test_auth.bats, line 53)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 9 create, config, api, and delete with basic auth # in 16669 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-21 08:52:37.184; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:52:37.188; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 10 post, postlogs and export with basic auth # in 16653 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/10-2
# Last output:
# WARN - 2026-05-21 08:52:53.859; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:52:53.862; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 11 package with basic auth # in 16683 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/11-3
# Last output:
# WARN - 2026-05-21 08:53:10.566; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:53:10.570; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 12 admin UI static content is excepted from basic auth # in 16665 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/12-4
# Last output:
# WARN - 2026-05-21 08:53:27.249; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:53:27.253; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 13 setup_file failed
# (from function `setup_file' in test file test/test_bats.bats, line 26)
# `solr start --verbose' failed
# Using Solr root directory: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT
# Using Java: /home/jenkins/tools/java/latest21/bin/java
# openjdk version "21.0.4" 2024-07-16 LTS
# OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
# OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
not ok 14 setup_file failed
# (from function `setup_file' in test file test/test_compression.bats, line 22)
# `solr start -e films' failed
#
# Starting up Solr on port 46519 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46519 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46519 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test # at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/-
not ok 17 setup_file failed
# (from function `setup_file' in test file test/test_config.bats, line 22)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
not ok 20 create for non cloud mode # in 1467 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 34)
# `assert_output --partial "Created new core 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created new core 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/20-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 21 create for cloud mode # in 1495 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 40)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/21-2
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
ok 22 multiple connection options are prevented # in 822 ms
not ok 23 setup_file failed
# (from function `setup_file' in test file test/test_create_collection.bats, line 22)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
not ok 34 setup_file failed
# (from function `setup_file' in test file test/test_delete_collection.bats, line 22)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
not ok 39 start -e cloud works with --jvm-opts # in 1305 ms
# (in test file test/test_example.bats, line 32)
# `solr start -e cloud --no-prompt --jvm-opts "-Dcustom.prop=helloworld"' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Creating Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr
# Cloning /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1 into
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2
#
# Starting up Solr on port 46519 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46519 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" --jvm-opts "-Dcustom.prop=helloworld"
#
#
# ERROR: Port 46519 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/39-1
not ok 40 start -e cloud works with --no-prompt # in 1299 ms
# (in test file test/test_example_noprompt.bats, line 32)
# `solr start -e cloud --no-prompt' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr already exists.
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2 already exists.
#
# Starting up Solr on port 46519 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46519 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46519 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
not ok 41 Check export command # in 2981 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_export.bats, line 36)
# `assert_output --partial 'Export complete'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Export complete
# output (48 lines):
# WARN - 2026-05-21 08:53:57.684; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:46519/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46519/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/41-1
# Last output:
# WARN - 2026-05-21 08:53:57.684; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:46519/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46519/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
ok 42 export fails on non cloud mode # in 2743 ms
Waiting for Tika Server to be ready on port 46524
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 43 using curl to extract a single pdf file # in 16620 ms
WARNING: Docker not available; bypassing test.
ok 44 using the bin/solr post tool to extract content from pdf # in 16582 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to crawl web site # in 16585 ms
Stopping Tika Server container
not ok 46 healthcheck on cloud solr # in 1323 ms
# (in test file test/test_healthcheck.bats, line 32)
# `solr start -e films' failed
#
# Starting up Solr on port 46519 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46519 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46519 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/46-1
not ok 47 healthcheck errors on standalone solr # in 1315 ms
# (in test file test/test_healthcheck.bats, line 39)
# `solr start --user-managed -e films' failed
#
# Starting up Solr on port 46519 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start --user-managed -p 46519 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 46519 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/47-2
ok 48 solr help flag prints help # in 795 ms
ok 49 solr with no flags prints help # in 748 ms
ok 50 start help flag prints help # in 151 ms
ok 51 start h flag prints help # in 159 ms
ok 52 stop help flag prints help # in 152 ms
ok 53 restart help flag prints help # in 154 ms
ok 54 status help flag prints help # in 809 ms
ok 55 healthcheck help flag prints help # in 766 ms
ok 56 create help flag prints help # in 787 ms
ok 57 delete help flag prints help # in 802 ms
ok 58 zk help flag prints help # in 847 ms
ok 59 auth help flag prints help # in 794 ms
ok 60 assert help flag prints help # in 803 ms
ok 61 post help flag prints help # in 780 ms
not ok 62 SQL Module # in 19050 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 36)
# `assert_output --partial '"docs":'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# "docs":
# output (16 lines):
#
# ERROR: Error from server at http://localhost:46519/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/62-1
# WARN - 2026-05-21 08:55:17.158; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:55:17.161; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-21 08:55:32.902; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# ERROR: Error from server at http://localhost:46519/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 63 icu collation in analysis-extras module # in 17941 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 49)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/63-2
# WARN - 2026-05-21 08:55:35.150; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:55:35.153; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-21 08:55:50.877; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 64 Check lifecycle of sentiment classification # in 11373 ms
# (in test file test/test_opennlp.bats, line 63)
# `solr start -m 4g -Dsolr.modules=analysis-extras -Dsolr.packages.enabled=true' failed
# Downloading onnx model and vocab...
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 160M 0 512k 0 0 378k 0 0:07:15 0:00:01 0:07:14 377k
8 160M 8 13.8M 0 0 6046k 0 0:00:27 0:00:02 0:00:25 6045k
21 160M 21 34.3M 0 0 10.2M 0 0:00:15 0:00:03 0:00:12 10.2M
33 160M 33 53.2M 0 0 12.2M 0 0:00:13 0:00:04 0:00:09 12.2M
45 160M 45 72.9M 0 0 13.5M 0 0:00:11 0:00:05 0:00:06 14.4M
57 160M 57 92.7M 0 0 14.5M 0 0:00:11 0:00:06 0:00:05 18.3M
69 160M 69 111M 0 0 15.1M 0 0:00:10 0:00:07 0:00:03 19.4M
80 160M 80 129M 0 0 15.5M 0 0:00:10 0:00:08 0:00:02 19.2M
92 160M 92 149M 0 0 16.0M 0 0:00:10 0:00:09 0:00:01 19.3M
100 160M 100 160M 0 0 16.3M 0 0:00:09 0:00:09 --:--:-- 19.6M
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 851k 100 851k 0 0 641k 0 0:00:01 0:00:01 --:--:-- 642k
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/64-1
# Last output:
# total 64M
# drwxr-xr-x 2 jenkins jenkins 4.0K May 21 08:41 .
# drwxr-xr-x 3 jenkins jenkins 4.0K May 21 08:41 ..
# -rw-r--r-- 1 jenkins jenkins 14M May 21 08:41 icu4j-77.1.jar
# -rw-r--r-- 1 jenkins jenkins 83K May 21 08:41 lucene-analysis-icu-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 30K May 21 08:41 lucene-analysis-morfologik-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 40K May 21 08:41 lucene-analysis-opennlp-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 3.5M May 21 08:41 lucene-analysis-smartcn-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 508K May 21 08:41 lucene-analysis-stempel-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 19K May 21 08:41 morfologik-fsa-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 1.8M May 21 08:41 morfologik-polish-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 51K May 21 08:41 morfologik-stemming-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 4.0M May 21 08:41 morfologik-ukrainian-search-4.9.1.jar
# -rw-r--r-- 1 jenkins jenkins 39M May 21 08:41 onnxruntime-1.24.3.jar
# -rw-r--r-- 1 jenkins jenkins 28K May 21 08:41 opennlp-dl-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 1.4M May 21 08:41 opennlp-tools-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 39K May 21 08:41 solr-analysis-extras-10.1.0-SNAPSHOT.jar
not ok 65 lifecycle of package # in 2333 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_packages.bats, line 38)
# `assert_output --partial "Available packages:"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Available packages:
# output (45 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/65-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 66 deploying and undeploying a collection level package # in 1356 ms
# (in test file test/test_packages.bats, line 44)
# `solr create -c foo-1.2' failed
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46519.
#
# ERROR: Error from server at http://localhost:46519/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:46519/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/66-2
# Last output:
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
not ok 68 Affinity placement plugin using sysprop # in 21752 ms
# (in test file test/test_placement_plugin.bats, line 34)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:46519 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/68-1
# WARN - 2026-05-21 08:56:12.534; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:56:12.537; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-21 08:56:28.263; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
not ok 69 Random placement plugin using ENV # in 21772 ms
# (in test file test/test_placement_plugin.bats, line 43)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:46519 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/69-2
# WARN - 2026-05-21 08:56:34.319; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-21 08:56:34.322; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-21 08:56:50.055; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
not ok 70 setup_file failed
# (from function `setup_file' in test file test/test_post.bats, line 22)
# `solr start -Dsolr.modules=extraction' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
not ok 81 setup_file failed
# (from function `setup_file' in test file test/test_postlogs.bats, line 22)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 16 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-21T08:56:57.571658252Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-21T08:56:57.571712536Z It should be set to 65000 to avoid operational disruption.
2026-05-21T08:56:57.571715985Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-21T08:56:57.572029254Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-21T08:56:57.572042732Z
2026-05-21T08:56:57.581837356Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-21T08:56:57.590465470Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-21T08:56:57.650778476Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-21T08:56:57.683130688Z WARNING: A command line option has enabled the Security Manager
2026-05-21T08:56:57.683156081Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-21T08:56:58.861549729Z 2026-05-21 08:56:58.858 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-21T08:56:59.140770255Z 2026-05-21 08:56:59.140 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-21T08:56:59.166627004Z 2026-05-21 08:56:59.166 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-21T08:56:59.173750996Z 2026-05-21 08:56:59.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-21T08:56:59.174145948Z 2026-05-21 08:56:59.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-21T08:56:59.174160178Z 2026-05-21 08:56:59.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-21T08:56:59.174163951Z 2026-05-21 08:56:59.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-21T08:56:59.173400226Z
2026-05-21T08:56:59.176380629Z 2026-05-21 08:56:59.175 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-21T08:56:59.197253763Z 2026-05-21 08:56:59.196 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-21T08:56:59.210235754Z 2026-05-21 08:56:59.209 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-21T08:56:59.210591261Z 2026-05-21 08:56:59.209 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-21T08:56:59.250499163Z 2026-05-21 08:56:59.249 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-21T08:57:00.037152309Z 2026-05-21 08:57:00.036 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-21T08:57:00.285093751Z 2026-05-21 08:57:00.284 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-21T08:57:00.290969653Z 2026-05-21 08:57:00.290 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-21T08:57:00.291157357Z 2026-05-21 08:57:00.290 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-21T08:57:00.294112556Z 2026-05-21 08:57:00.293 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-21T08:57:00.294127468Z 2026-05-21 08:57:00.293 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-21T08:57:00.335654014Z 2026-05-21 08:57:00.335 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-21T08:57:00.796335816Z 2026-05-21 08:57:00.795 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-21T08:57:00.809078525Z 2026-05-21 08:57:00.808 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-21T08:57:00.850885005Z 2026-05-21 08:57:00.850 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-21T08:57:00.875136108Z 2026-05-21 08:57:00.874 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-21T08:57:00.875276863Z 2026-05-21 08:57:00.874 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-21T08:57:01.015577945Z 2026-05-21 08:57:01.014 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-21T08:57:01.039137982Z 2026-05-21 08:57:01.038 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-21T08:57:01.050633379Z 2026-05-21 08:57:01.050 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-21T08:57:01.056876804Z 2026-05-21 08:57:01.056 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72092193407959040-solr-node1:8983_solr-n_0000000000) starting
2026-05-21T08:57:01.123396989Z 2026-05-21 08:57:01.122 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-21T08:57:01.124318924Z 2026-05-21 08:57:01.123 INFO (OverseerStateUpdate-72092193407959040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-21T08:57:01.133353914Z 2026-05-21 08:57:01.132 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-21T08:57:01.173471325Z 2026-05-21 08:57:01.172 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-21T08:57:01.544525227Z 2026-05-21 08:57:01.543 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-21T08:57:01.727365288Z 2026-05-21 08:57:01.726 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-21T08:57:02.061734910Z 2026-05-21 08:57:02.061 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-21T08:57:02.334944530Z 2026-05-21 08:57:02.334 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-21T08:57:02.406982030Z 2026-05-21 08:57:02.405 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_21.request.log
2026-05-21T08:57:02.411163182Z 2026-05-21 08:57:02.410 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-21T08:57:02.412742192Z 2026-05-21 08:57:02.412 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4824ms
2026-05-21T08:57:02.701331609Z 2026-05-21 08:57:02.700 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=47
2026-05-21T08:57:02.830570026Z 2026-05-21 08:57:02.830 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-05-21T08:57:06.127792265Z 2026-05-21 08:57:06.127 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-21T08:57:06.304416521Z 2026-05-21 08:57:06.303 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-21T08:57:10.144341349Z 2026-05-21 08:57:10.143 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=42
2026-05-21T08:57:10.204235217Z 2026-05-21 08:57:10.203 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=40
2026-05-21T08:57:10.234897408Z 2026-05-21 08:57:10.234 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=4
2026-05-21T08:57:10.837908752Z 2026-05-21 08:57:10.836 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-21T08:57:10.865216713Z 2026-05-21 08:57:10.864 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-21T08:57:10.884509640Z 2026-05-21 08:57:10.884 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-21T08:57:10.958131630Z 2026-05-21 08:57:10.957 INFO (OverseerStateUpdate-72092193407959040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-21T08:57:10.958151462Z "core":"test-collection_shard1_replica_n1",
2026-05-21T08:57:10.958154547Z "node_name":"solr-node2:8984_solr",
2026-05-21T08:57:10.958156804Z "base_url":"http://solr-node2:8984/solr",
2026-05-21T08:57:10.958159126Z "collection":"test-collection",
2026-05-21T08:57:10.958161260Z "shard":"shard1",
2026-05-21T08:57:10.958164924Z "state":"down",
2026-05-21T08:57:10.958167536Z "type":"NRT",
2026-05-21T08:57:10.958169486Z "operation":"ADDREPLICA",
2026-05-21T08:57:10.958171423Z "waitForFinalState":"false"}
2026-05-21T08:57:10.966633337Z 2026-05-21 08:57:10.966 INFO (OverseerStateUpdate-72092193407959040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-21T08:57:10.966659930Z "core":"test-collection_shard2_replica_n2",
2026-05-21T08:57:10.966664732Z "node_name":"solr-node3:8985_solr",
2026-05-21T08:57:10.966686728Z "base_url":"http://solr-node3:8985/solr",
2026-05-21T08:57:10.966692113Z "collection":"test-collection",
2026-05-21T08:57:10.966695081Z "shard":"shard2",
2026-05-21T08:57:10.966697866Z "state":"down",
2026-05-21T08:57:10.966700368Z "type":"NRT",
2026-05-21T08:57:10.966703931Z "operation":"ADDREPLICA",
2026-05-21T08:57:10.966706780Z "waitForFinalState":"false"}
2026-05-21T08:57:10.973893357Z 2026-05-21 08:57:10.973 INFO (OverseerStateUpdate-72092193407959040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-21T08:57:10.973910966Z "core":"test-collection_shard3_replica_n4",
2026-05-21T08:57:10.973914397Z "node_name":"solr-node1:8983_solr",
2026-05-21T08:57:10.973926208Z "base_url":"http://solr-node1:8983/solr",
2026-05-21T08:57:10.973928372Z "collection":"test-collection",
2026-05-21T08:57:10.973931319Z "shard":"shard3",
2026-05-21T08:57:10.973933272Z "state":"down",
2026-05-21T08:57:10.973935188Z "type":"NRT",
2026-05-21T08:57:10.973937157Z "operation":"ADDREPLICA",
2026-05-21T08:57:10.973939112Z "waitForFinalState":"false"}
2026-05-21T08:57:11.085648237Z 2026-05-21 08:57:11.084 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-21T08:57:11.154712388Z 2026-05-21 08:57:11.153 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-21T08:57:11.206330026Z 2026-05-21 08:57:11.205 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-21T08:57:11.253817594Z 2026-05-21 08:57:11.253 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-21T08:57:11.647484677Z 2026-05-21 08:57:11.645 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-21T08:57:11.647555449Z 2026-05-21 08:57:11.645 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-21T08:57:11.671336334Z 2026-05-21 08:57:11.670 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n4], dataDir=[/var/solr/data/test-collection_shard3_replica_n4/data/]
2026-05-21T08:57:12.875081487Z 2026-05-21 08:57:12.873 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-21T08:57:12.937974742Z 2026-05-21 08:57:12.936 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-21T08:57:12.946126191Z 2026-05-21 08:57:12.945 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-21T08:57:12.948180106Z 2026-05-21 08:57:12.947 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-21T08:57:12.948223473Z 2026-05-21 08:57:12.947 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-21T08:57:13.085623966Z 2026-05-21 08:57:13.084 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-21T08:57:13.092305062Z 2026-05-21 08:57:13.091 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-21T08:57:13.095866746Z 2026-05-21 08:57:13.095 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-21T08:57:13.098626703Z 2026-05-21 08:57:13.098 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-21T08:57:13.156102730Z 2026-05-21 08:57:13.155 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-21T08:57:13.164008883Z 2026-05-21 08:57:13.163 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-21T08:57:13.170768095Z 2026-05-21 08:57:13.170 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-21T08:57:13.170787955Z 2026-05-21 08:57:13.170 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-21T08:57:13.240042258Z 2026-05-21 08:57:13.239 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-21T08:57:13.248849673Z 2026-05-21 08:57:13.247 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-21T08:57:13.255383625Z 2026-05-21 08:57:13.254 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-21T08:57:13.271535263Z 2026-05-21 08:57:13.271 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-21T08:57:13.271565257Z 2026-05-21 08:57:13.271 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-21T08:57:13.271645365Z 2026-05-21 08:57:13.271 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n4/
2026-05-21T08:57:13.272973246Z 2026-05-21 08:57:13.272 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-21T08:57:13.273401842Z 2026-05-21 08:57:13.273 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ has no replicas
2026-05-21T08:57:13.274501862Z 2026-05-21 08:57:13.274 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72092193407959040-core_node6-n_0000000000
2026-05-21T08:57:13.281868293Z 2026-05-21 08:57:13.281 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ shard3
2026-05-21T08:57:13.387113452Z 2026-05-21 08:57:13.386 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-21T08:57:13.391341407Z 2026-05-21 08:57:13.391 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-21T08:57:13.400087613Z 2026-05-21 08:57:13.399 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2248
2026-05-21T08:57:13.500060179Z 2026-05-21 08:57:13.499 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 248] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1332815,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-21T08:56:57.353845593Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-21T08:57:03.155515056Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-21T08:57:03.155547316Z It should be set to 65000 to avoid operational disruption.
2026-05-21T08:57:03.155551185Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-21T08:57:03.155905331Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-21T08:57:03.155916156Z
2026-05-21T08:57:03.165850288Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-21T08:57:03.173323952Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-21T08:57:03.227416386Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-21T08:57:03.268785199Z WARNING: A command line option has enabled the Security Manager
2026-05-21T08:57:03.268803694Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-21T08:57:04.567818363Z 2026-05-21 08:57:04.564 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-21T08:57:04.844454722Z 2026-05-21 08:57:04.843 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-21T08:57:04.869681412Z 2026-05-21 08:57:04.869 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-21T08:57:04.876998650Z 2026-05-21 08:57:04.876 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-21T08:57:04.877053439Z 2026-05-21 08:57:04.876 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-21T08:57:04.877216189Z 2026-05-21 08:57:04.876 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-21T08:57:04.877332532Z 2026-05-21 08:57:04.876 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-21T08:57:04.876763113Z
2026-05-21T08:57:04.879873173Z 2026-05-21 08:57:04.879 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-21T08:57:04.903721002Z 2026-05-21 08:57:04.903 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-21T08:57:04.932281780Z 2026-05-21 08:57:04.931 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-21T08:57:05.077244957Z 2026-05-21 08:57:05.076 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-21T08:57:05.096203737Z 2026-05-21 08:57:05.095 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-21T08:57:05.096298118Z 2026-05-21 08:57:05.095 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-21T08:57:05.096452972Z 2026-05-21 08:57:05.095 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-21T08:57:05.206145509Z 2026-05-21 08:57:05.205 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-21T08:57:05.206270417Z 2026-05-21 08:57:05.205 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-21T08:57:05.206437626Z 2026-05-21 08:57:05.206 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-21T08:57:05.241570114Z 2026-05-21 08:57:05.241 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-21T08:57:06.026352052Z 2026-05-21 08:57:06.025 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-21T08:57:06.178969334Z 2026-05-21 08:57:06.178 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-21T08:57:06.180900670Z 2026-05-21 08:57:06.178 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-21T08:57:06.191756993Z 2026-05-21 08:57:06.191 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-21T08:57:06.197052417Z 2026-05-21 08:57:06.196 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-21T08:57:06.201833236Z 2026-05-21 08:57:06.201 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-21T08:57:06.201854574Z 2026-05-21 08:57:06.201 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-21T08:57:06.240269772Z 2026-05-21 08:57:06.239 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-21T08:57:06.249866872Z 2026-05-21 08:57:06.249 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-21T08:57:06.273441837Z 2026-05-21 08:57:06.272 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-21T08:57:06.286008017Z 2026-05-21 08:57:06.285 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-21T08:57:06.296175502Z 2026-05-21 08:57:06.295 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-21T08:57:06.305478117Z 2026-05-21 08:57:06.304 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-21T08:57:06.352221247Z 2026-05-21 08:57:06.351 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-21T08:57:06.638046181Z 2026-05-21 08:57:06.637 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-21T08:57:06.810384537Z 2026-05-21 08:57:06.809 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-21T08:57:07.096762216Z 2026-05-21 08:57:07.096 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-21T08:57:07.237460181Z 2026-05-21 08:57:07.236 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-21T08:57:07.242823318Z 2026-05-21 08:57:07.242 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_21.request.log
2026-05-21T08:57:07.246021266Z 2026-05-21 08:57:07.245 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-21T08:57:07.246954828Z 2026-05-21 08:57:07.246 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4074ms
2026-05-21T08:57:07.404078498Z 2026-05-21 08:57:07.403 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=45
2026-05-21T08:57:07.462255901Z 2026-05-21 08:57:07.461 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-05-21T08:57:11.151244808Z 2026-05-21 08:57:11.150 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard1_replica_n1&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-21T08:57:11.205941757Z 2026-05-21 08:57:11.205 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-21T08:57:11.275041608Z 2026-05-21 08:57:11.274 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-21T08:57:11.666958666Z 2026-05-21 08:57:11.666 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-21T08:57:11.667051166Z 2026-05-21 08:57:11.666 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-21T08:57:11.687814217Z 2026-05-21 08:57:11.687 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n1], dataDir=[/var/solr/data/test-collection_shard1_replica_n1/data/]
2026-05-21T08:57:12.147704749Z 2026-05-21 08:57:12.147 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-21T08:57:12.169102946Z 2026-05-21 08:57:12.168 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-21T08:57:12.173076232Z 2026-05-21 08:57:12.172 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-21T08:57:12.174317632Z 2026-05-21 08:57:12.173 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-21T08:57:12.174335157Z 2026-05-21 08:57:12.173 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-21T08:57:12.216039289Z 2026-05-21 08:57:12.215 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-21T08:57:12.219327312Z 2026-05-21 08:57:12.219 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-21T08:57:12.221595099Z 2026-05-21 08:57:12.221 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-21T08:57:12.223598000Z 2026-05-21 08:57:12.223 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-21T08:57:12.236413325Z 2026-05-21 08:57:12.235 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-21T08:57:12.245808473Z 2026-05-21 08:57:12.245 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-21T08:57:12.253307538Z 2026-05-21 08:57:12.252 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node3 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-21T08:57:12.253326160Z 2026-05-21 08:57:12.252 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node3 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-21T08:57:12.259542492Z 2026-05-21 08:57:12.259 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node3 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-21T08:57:12.2737
[...truncated too long message...]
igned certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Certificate was added to keystore
# Certificate was added to keystore
# Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Certificate was added to keystore
# Certificate was added to keystore
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/92-6
# Last output:
# INFO - 2026-05-21 08:58:15.611; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-05-21 08:58:15.614; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-05-21 08:58:15.625; org.apache.solr.common.util.EnvUtils; You are passing in deprecated system property host and should upgrade to using solr.host.advertise instead. The deprecated property support will be removed in future version of Solr.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is https://localhost:46519.
#
# ERROR: IOException occurred when talking to server at: https://localhost:46519/solr/admin/info/system
not ok 93 test keystore reload # in 2709 ms
# (in test file test/test_ssl.bats, line 529)
# `solr start --jvm-opts "-Dsolr.jetty.ssl.context.reload.scan.interval.secs=1 -DsocketTimeout=5000"' failed
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/93-7
# Last output:
# INFO - 2026-05-21 08:58:18.338; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-05-21 08:58:18.341; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-05-21 08:58:18.352; org.apache.solr.common.util.EnvUtils; You are passing in deprecated system property host and should upgrade to using solr.host.advertise instead. The deprecated property support will be removed in future version of Solr.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is https://localhost:46519.
#
# ERROR: IOException occurred when talking to server at: https://localhost:46519/solr/admin/info/system
not ok 94 SOLR-11740 check 'solr stop' connection # in 244 ms
# (in test file test/test_start_solr.bats, line 32)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/94-1
not ok 95 stop command for single port # in 272 ms
# (in test file test/test_start_solr.bats, line 45)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/95-2
ok 96 check stop command doesn't hang # in 240 ms
not ok 97 SOLR-16976 solr starts with remote JMX enabled # in 250 ms
# (in test file test/test_start_solr.bats, line 77)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/97-4
not ok 98 deprecated system properties converted to modern properties # in 250 ms
# (in test file test/test_start_solr.bats, line 85)
# `solr start -Ddisable.config.edit=true' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/98-5
not ok 99 start with custom jetty options # in 247 ms
# (in test file test/test_start_solr.bats, line 93)
# `solr start --jettyconfig "--module=server"' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/99-6
not ok 100 bootstrapping a configset # in 243 ms
# (in test file test/test_start_solr.bats, line 104)
# `solr start -Dbootstrap_confdir="${confdir_path}" -Dcollection.configName=techproducts' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/100-7
not ok 101 status detects locally running solr # in 1018 ms
# (in test file test/test_status.bats, line 34)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/101-1
# Last output:
#
# No Solr nodes are running.
not ok 102 status with --solr-url from user # in 228 ms
# (in test file test/test_status.bats, line 43)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/102-2
not ok 103 status with --port from user # in 230 ms
# (in test file test/test_status.bats, line 50)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-3
ok 104 multiple connection options are prevented # in 894 ms
ok 105 status with invalid --solr-url from user # in 1301 ms
not ok 106 status with --short format # in 234 ms
# (in test file test/test_status.bats, line 67)
# `solr start' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-6
not ok 107 setup_file failed
# (from function `setup_file' in test file test/test_stream.bats, line 22)
# `solr start -e techproducts' failed
#
# Starting up Solr on port 46519 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 46519 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" -Dsolr.modules=clustering,extraction,langid,ltr,scripting -Dsolr.ltr.enabled=true -Dsolr.clustering.enabled=true
#
#
# ERROR: Port 46519 is already being used by another process.
#
ok 111 --version returns Solr version # in 773 ms
ok 112 version as direct tool call still runs # in 770 ms
not ok 113 setup_file failed
# (from function `setup_file' in test file test/test_zk.bats, line 22)
# `solr start -DminStateByteLenForCompression=0' failed
#
# Port 46519 is already being used by another process (pid: 1324609)
# Please choose a different port using the -p option.
#
ok 125 Cleanup # in 1899 ms
Build timed out (after 1,217 minutes). Marking the build as aborted.
Build timed out (after 1,217 minutes). Marking the build as failed.
Build was aborted
Archiving artifacts
Terminated
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 214 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/214/
No tests ran.
Build Log:
[...truncated 372 lines...]
# ERROR: Solr is not running on url http://localhost:35101 after 5 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/4-2
# Last output:
# Waiting up to 180 seconds to see Solr running on port 35101 [|] [/] [-] [\] [|]
# Started Solr server on port 35101 (pid=1188051). Happy searching!
#
#
ok 5 auth rejects block-unknown option with invalid boolean # in 1780 ms
ok 6 auth rejects update-include-file-only option with invalid boolean # in 1768 ms
not ok 7 auth enable/disable lifecycle # in 20618 ms
# (in test file test/test_auth.bats, line 40)
# `solr auth enable --type basicAuth --credentials name:password' failed
# Waiting up to 180 seconds to see Solr running on port 35101 [|] [/] [-] [\] [|]
# Started Solr server on port 35101 (pid=1192605). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
# Couldn't get ZooKeeper host. Please make sure Solr is running in cloud mode, or a zk-host has been passed in.
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Server refused connection at: http://localhost:35101/solr/admin/info/system
not ok 8 enable auth connects via zookeeper # in 1216 ms
# (in test file test/test_auth.bats, line 53)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 9 create, config, api, and delete with basic auth # in 16635 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-19 22:08:10.193; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:08:10.196; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 10 post, postlogs and export with basic auth # in 16615 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/10-2
# Last output:
# WARN - 2026-05-19 22:08:26.837; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:08:26.840; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 11 package with basic auth # in 16646 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/11-3
# Last output:
# WARN - 2026-05-19 22:08:43.497; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:08:43.500; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 12 admin UI static content is excepted from basic auth # in 16654 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/12-4
# Last output:
# WARN - 2026-05-19 22:09:00.166; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:09:00.169; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
not ok 13 setup_file failed
# (from function `setup_file' in test file test/test_bats.bats, line 26)
# `solr start --verbose' failed
# Using Solr root directory: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT
# Using Java: /home/jenkins/tools/java/latest21/bin/java
# openjdk version "21.0.4" 2024-07-16 LTS
# OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
# OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
not ok 14 setup_file failed
# (from function `setup_file' in test file test/test_compression.bats, line 22)
# `solr start -e films' failed
#
# Starting up Solr on port 35101 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 35101 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 35101 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test # at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/-
not ok 17 setup_file failed
# (from function `setup_file' in test file test/test_config.bats, line 22)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
not ok 20 create for non cloud mode # in 1424 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 34)
# `assert_output --partial "Created new core 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created new core 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/20-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 21 create for cloud mode # in 1387 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_create.bats, line 40)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/21-2
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
ok 22 multiple connection options are prevented # in 808 ms
not ok 23 setup_file failed
# (from function `setup_file' in test file test/test_create_collection.bats, line 22)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
not ok 34 setup_file failed
# (from function `setup_file' in test file test/test_delete_collection.bats, line 22)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
not ok 39 start -e cloud works with --jvm-opts # in 1223 ms
# (in test file test/test_example.bats, line 32)
# `solr start -e cloud --no-prompt --jvm-opts "-Dcustom.prop=helloworld"' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Creating Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr
# Cloning /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1 into
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2
#
# Starting up Solr on port 35101 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 35101 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" --jvm-opts "-Dcustom.prop=helloworld"
#
#
# ERROR: Port 35101 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/39-1
not ok 40 start -e cloud works with --no-prompt # in 1236 ms
# (in test file test/test_example_noprompt.bats, line 32)
# `solr start -e cloud --no-prompt' failed
#
# Welcome to the SolrCloud example!
#
# Starting up 2 Solr nodes for your example SolrCloud cluster.
#
# Solr home directory /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr already exists.
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/example/cloud/node2 already exists.
#
# Starting up Solr on port 35101 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 35101 --solr-home "build/solr-10.1.0-SNAPSHOT/example/cloud/node1/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 35101 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
not ok 41 Check export command # in 2865 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_export.bats, line 36)
# `assert_output --partial 'Export complete'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Export complete
# output (48 lines):
# WARN - 2026-05-19 22:09:29.988; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:35101/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:35101/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/41-1
# Last output:
# WARN - 2026-05-19 22:09:29.988; org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider; Attempt to fetch live_nodes from http://localhost:35101/solr failed.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:35101/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin: Expected mime type in [application/vnd.apache.solr.javabin, application/octet-stream] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/collections?action=CLUSTERSTATUS&includeAll=false&liveNodes=true&prs=true&wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1194)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.submitClusterStateRequest(BaseHttpClusterStateProvider.java:221)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:293)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.updateLiveNodes(BaseHttpClusterStateProvider.java:280)
# at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
# at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
# at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
# at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
# at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
# at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
# at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
# at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
# at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:632)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:268)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getLiveNodes(BaseHttpClusterStateProvider.java:231)
# at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.getState(BaseHttpClusterStateProvider.java:109)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionRef(CloudSolrClient.java:1723)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.loadDocCollection(CloudSolrClient.java:1699)
# at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$triggerCollectionRefresh$6(CloudSolrClient.java:1664)
# at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
#
# ERROR: Error refreshing state for collection techproducts
ok 42 export fails on non cloud mode # in 2699 ms
Waiting for Tika Server to be ready on port 35106
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 43 using curl to extract a single pdf file # in 16589 ms
WARNING: Docker not available; bypassing test.
ok 44 using the bin/solr post tool to extract content from pdf # in 16539 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to crawl web site # in 16516 ms
Stopping Tika Server container
not ok 46 healthcheck on cloud solr # in 1228 ms
# (in test file test/test_healthcheck.bats, line 32)
# `solr start -e films' failed
#
# Starting up Solr on port 35101 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 35101 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 35101 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/46-1
not ok 47 healthcheck errors on standalone solr # in 1254 ms
# (in test file test/test_healthcheck.bats, line 39)
# `solr start --user-managed -e films' failed
#
# Starting up Solr on port 35101 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start --user-managed -p 35101 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server"
#
#
# ERROR: Port 35101 is already being used by another process.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/47-2
ok 48 solr help flag prints help # in 714 ms
ok 49 solr with no flags prints help # in 717 ms
ok 50 start help flag prints help # in 143 ms
ok 51 start h flag prints help # in 143 ms
ok 52 stop help flag prints help # in 142 ms
ok 53 restart help flag prints help # in 144 ms
ok 54 status help flag prints help # in 784 ms
ok 55 healthcheck help flag prints help # in 737 ms
ok 56 create help flag prints help # in 731 ms
ok 57 delete help flag prints help # in 745 ms
ok 58 zk help flag prints help # in 754 ms
ok 59 auth help flag prints help # in 730 ms
ok 60 assert help flag prints help # in 741 ms
ok 61 post help flag prints help # in 747 ms
not ok 62 SQL Module # in 18841 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 36)
# `assert_output --partial '"docs":'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# "docs":
# output (16 lines):
#
# ERROR: Error from server at http://localhost:35101/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/62-1
# WARN - 2026-05-19 22:10:37.958; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:10:37.961; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-19 22:10:53.686; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# ERROR: Error from server at http://localhost:35101/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json: Expected mime type in [application/json] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/COLL_NAME/sql?stmt=select+id+from+COLL_NAME+limit+10&wt=json</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 63 icu collation in analysis-extras module # in 17798 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_modules.bats, line 49)
# `assert_output --partial "Created collection 'COLL_NAME'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'COLL_NAME'
# output (17 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/63-2
# WARN - 2026-05-19 22:10:55.784; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:10:55.787; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 64 Check lifecycle of sentiment classification # in 11468 ms
# (in test file test/test_opennlp.bats, line 63)
# `solr start -m 4g -Dsolr.modules=analysis-extras -Dsolr.packages.enabled=true' failed
# Downloading onnx model and vocab...
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 160M 0 112k 0 0 108k 0 0:25:16 0:00:01 0:25:15 108k
4 160M 4 7536k 0 0 3758k 0 0:00:43 0:00:02 0:00:41 3760k
17 160M 17 27.5M 0 0 9359k 0 0:00:17 0:00:03 0:00:14 9359k
29 160M 29 46.6M 0 0 11.6M 0 0:00:13 0:00:04 0:00:09 11.6M
41 160M 41 66.5M 0 0 13.2M 0 0:00:12 0:00:05 0:00:07 13.2M
53 160M 53 86.7M 0 0 14.3M 0 0:00:11 0:00:06 0:00:05 17.3M
65 160M 65 105M 0 0 15.0M 0 0:00:10 0:00:07 0:00:03 19.6M
78 160M 78 125M 0 0 15.6M 0 0:00:10 0:00:08 0:00:02 19.6M
89 160M 89 144M 0 0 15.9M 0 0:00:10 0:00:09 0:00:01 19.4M
100 160M 100 160M 0 0 16.3M 0 0:00:09 0:00:09 --:--:-- 19.4M
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
24 851k 24 208k 0 0 176k 0 0:00:04 0:00:01 0:00:03 175k
100 851k 100 851k 0 0 593k 0 0:00:01 0:00:01 --:--:-- 593k
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/64-1
# Last output:
# total 64M
# drwxr-xr-x 2 jenkins jenkins 4.0K May 19 21:56 .
# drwxr-xr-x 3 jenkins jenkins 4.0K May 19 21:56 ..
# -rw-r--r-- 1 jenkins jenkins 14M May 19 21:56 icu4j-77.1.jar
# -rw-r--r-- 1 jenkins jenkins 83K May 19 21:56 lucene-analysis-icu-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 30K May 19 21:56 lucene-analysis-morfologik-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 40K May 19 21:56 lucene-analysis-opennlp-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 3.5M May 19 21:56 lucene-analysis-smartcn-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 508K May 19 21:56 lucene-analysis-stempel-10.4.0.jar
# -rw-r--r-- 1 jenkins jenkins 19K May 19 21:56 morfologik-fsa-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 1.8M May 19 21:56 morfologik-polish-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 51K May 19 21:56 morfologik-stemming-2.1.9.jar
# -rw-r--r-- 1 jenkins jenkins 4.0M May 19 21:56 morfologik-ukrainian-search-4.9.1.jar
# -rw-r--r-- 1 jenkins jenkins 39M May 19 21:56 onnxruntime-1.24.3.jar
# -rw-r--r-- 1 jenkins jenkins 28K May 19 21:56 opennlp-dl-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 1.4M May 19 21:56 opennlp-tools-2.5.8.jar
# -rw-r--r-- 1 jenkins jenkins 39K May 19 21:56 solr-analysis-extras-10.1.0-SNAPSHOT.jar
not ok 65 lifecycle of package # in 2174 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_packages.bats, line 38)
# `assert_output --partial "Available packages:"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Available packages:
# output (45 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/65-1
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
# org.apache.solr.client.solrj.RemoteSolrException: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
# at org.apache.solr.client.solrj.impl.HttpSolrClient.checkContentType(HttpSolrClient.java:308)
# at org.apache.solr.client.solrj.impl.HttpSolrClient.processErrorsAndResponse(HttpSolrClient.java:228)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.processErrorsAndResponse(HttpJettySolrClient.java:595)
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:489)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:348)
# at org.apache.solr.cli.StatusTool.reportStatus(StatusTool.java:300)
# at org.apache.solr.cli.CLIUtils.getZkHost(CLIUtils.java:248)
# at org.apache.solr.cli.PackageTool.runImpl(PackageTool.java:123)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
not ok 66 deploying and undeploying a collection level package # in 1274 ms
# (in test file test/test_packages.bats, line 44)
# `solr create -c foo-1.2' failed
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35101.
#
# ERROR: Error from server at http://localhost:35101/solr/admin/info/system?wt=javabin: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html>
# <head>
# <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
# <title>Error 503 Service Unavailable</title>
# </head>
# <body>
# <h2>HTTP ERROR 503 Service Unavailable</h2>
# <table>
# <tr><th>URI:</th><td>http://localhost:35101/solr/admin/info/system?wt=javabin</td></tr>
# <tr><th>STATUS:</th><td>503</td></tr>
# <tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
# </table>
#
# </body>
# </html>
#
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/66-2
# Last output:
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
not ok 68 Affinity placement plugin using sysprop # in 21752 ms
# (in test file test/test_placement_plugin.bats, line 34)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:35101 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/68-1
# WARN - 2026-05-19 22:11:32.951; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:11:32.954; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# ERROR - 2026-05-19 22:11:48.687; org.apache.solr.cli.ZkLsTool; Could not complete ls operation for reason:
# org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:213)
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:109)
# at org.apache.solr.common.cloud.SolrZkClient$Builder.build(SolrZkClient.java:1188)
# at org.apache.solr.cli.CLIUtils.getSolrZkClient(CLIUtils.java:273)
# at org.apache.solr.cli.ZkLsTool.runImpl(ZkLsTool.java:58)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
# at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:203)
# ... 6 more
# Last output:
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
not ok 69 Random placement plugin using ENV # in 21788 ms
# (in test file test/test_placement_plugin.bats, line 43)
# `solr assert --cloud http://localhost:${SOLR_PORT} --timeout 3000' failed with status 100
#
# ERROR: Solr is not running on url http://localhost:35101 after 3 seconds
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/69-2
# WARN - 2026-05-19 22:11:54.775; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-19 22:11:54.778; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
#
# Last output:
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
not ok 70 setup_file failed
# (from function `setup_file' in test file test/test_post.bats, line 22)
# `solr start -Dsolr.modules=extraction' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
not ok 81 setup_file failed
# (from function `setup_file' in test file test/test_postlogs.bats, line 22)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 16 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-19T22:12:20.650402259Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-19T22:12:20.650439705Z It should be set to 65000 to avoid operational disruption.
2026-05-19T22:12:20.650443052Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-19T22:12:20.650756848Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-19T22:12:20.650767947Z
2026-05-19T22:12:20.660161603Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-19T22:12:20.668316661Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-19T22:12:20.715530731Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-19T22:12:20.747390613Z WARNING: A command line option has enabled the Security Manager
2026-05-19T22:12:20.747413929Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-19T22:12:21.960701095Z 2026-05-19 22:12:21.957 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-19T22:12:22.229809982Z 2026-05-19 22:12:22.229 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-19T22:12:22.255431166Z 2026-05-19 22:12:22.254 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-19T22:12:22.262756937Z 2026-05-19 22:12:22.262 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-19T22:12:22.262893482Z 2026-05-19 22:12:22.262 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-19T22:12:22.262980795Z 2026-05-19 22:12:22.262 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-19T22:12:22.263140984Z 2026-05-19 22:12:22.262 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-19T22:12:22.262525981Z
2026-05-19T22:12:22.265819503Z 2026-05-19 22:12:22.265 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-19T22:12:22.288571097Z 2026-05-19 22:12:22.288 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-19T22:12:22.301706382Z 2026-05-19 22:12:22.301 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-19T22:12:22.301780631Z 2026-05-19 22:12:22.301 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-19T22:12:22.336539438Z 2026-05-19 22:12:22.335 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-19T22:12:23.253447544Z 2026-05-19 22:12:23.252 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-19T22:12:23.486222181Z 2026-05-19 22:12:23.485 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-19T22:12:23.491062067Z 2026-05-19 22:12:23.490 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-19T22:12:23.491171915Z 2026-05-19 22:12:23.490 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-19T22:12:23.493683999Z 2026-05-19 22:12:23.493 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-19T22:12:23.493857809Z 2026-05-19 22:12:23.493 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-19T22:12:23.528167890Z 2026-05-19 22:12:23.527 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-19T22:12:23.997662455Z 2026-05-19 22:12:23.996 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-19T22:12:24.011465276Z 2026-05-19 22:12:24.011 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-19T22:12:24.047821438Z 2026-05-19 22:12:24.047 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-19T22:12:24.139315218Z 2026-05-19 22:12:24.138 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-19T22:12:24.139442391Z 2026-05-19 22:12:24.138 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-19T22:12:24.270256052Z 2026-05-19 22:12:24.269 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-19T22:12:24.293999262Z 2026-05-19 22:12:24.293 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-19T22:12:24.306384587Z 2026-05-19 22:12:24.305 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-19T22:12:24.312547596Z 2026-05-19 22:12:24.312 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72083928546934784-solr-node1:8983_solr-n_0000000000) starting
2026-05-19T22:12:24.378155556Z 2026-05-19 22:12:24.377 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-19T22:12:24.379465990Z 2026-05-19 22:12:24.379 INFO (OverseerStateUpdate-72083928546934784-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-19T22:12:24.388303930Z 2026-05-19 22:12:24.387 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-19T22:12:24.430201594Z 2026-05-19 22:12:24.429 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-19T22:12:24.806071611Z 2026-05-19 22:12:24.803 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-19T22:12:24.982754690Z 2026-05-19 22:12:24.982 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-19T22:12:25.246245520Z 2026-05-19 22:12:25.245 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-19T22:12:25.377732296Z 2026-05-19 22:12:25.376 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-19T22:12:25.382977736Z 2026-05-19 22:12:25.382 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_19.request.log
2026-05-19T22:12:25.387124477Z 2026-05-19 22:12:25.386 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-19T22:12:25.388758728Z 2026-05-19 22:12:25.388 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4722ms
2026-05-19T22:12:25.773981575Z 2026-05-19 22:12:25.772 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=139
2026-05-19T22:12:25.851373628Z 2026-05-19 22:12:25.850 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-05-19T22:12:29.120638201Z 2026-05-19 22:12:29.120 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-19T22:12:29.282099130Z 2026-05-19 22:12:29.281 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-19T22:12:32.945220271Z 2026-05-19 22:12:32.944 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=42
2026-05-19T22:12:32.974500448Z 2026-05-19 22:12:32.973 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=10
2026-05-19T22:12:33.001126228Z 2026-05-19 22:12:33.000 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-19T22:12:33.240084618Z 2026-05-19 22:12:33.239 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=5
2026-05-19T22:12:33.268132262Z 2026-05-19 22:12:33.267 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-19T22:12:33.287399109Z 2026-05-19 22:12:33.286 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-19T22:12:33.293921422Z 2026-05-19 22:12:33.293 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-19T22:12:33.338977784Z 2026-05-19 22:12:33.338 INFO (OverseerStateUpdate-72083928546934784-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-19T22:12:33.338999437Z "core":"test-collection_shard3_replica_n1",
2026-05-19T22:12:33.339002014Z "node_name":"solr-node1:8983_solr",
2026-05-19T22:12:33.339004732Z "base_url":"http://solr-node1:8983/solr",
2026-05-19T22:12:33.339006848Z "collection":"test-collection",
2026-05-19T22:12:33.339008911Z "shard":"shard3",
2026-05-19T22:12:33.339010877Z "state":"down",
2026-05-19T22:12:33.339012893Z "type":"NRT",
2026-05-19T22:12:33.339014789Z "operation":"ADDREPLICA",
2026-05-19T22:12:33.339016919Z "waitForFinalState":"false"}
2026-05-19T22:12:33.345043975Z 2026-05-19 22:12:33.344 INFO (OverseerStateUpdate-72083928546934784-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-19T22:12:33.345060485Z "core":"test-collection_shard2_replica_n2",
2026-05-19T22:12:33.345062953Z "node_name":"solr-node3:8985_solr",
2026-05-19T22:12:33.345065153Z "base_url":"http://solr-node3:8985/solr",
2026-05-19T22:12:33.345067179Z "collection":"test-collection",
2026-05-19T22:12:33.345069216Z "shard":"shard2",
2026-05-19T22:12:33.345071243Z "state":"down",
2026-05-19T22:12:33.345073125Z "type":"NRT",
2026-05-19T22:12:33.345075149Z "operation":"ADDREPLICA",
2026-05-19T22:12:33.345077143Z "waitForFinalState":"false"}
2026-05-19T22:12:33.349220674Z 2026-05-19 22:12:33.348 INFO (OverseerStateUpdate-72083928546934784-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-19T22:12:33.349245585Z "core":"test-collection_shard1_replica_n4",
2026-05-19T22:12:33.349248382Z "node_name":"solr-node2:8984_solr",
2026-05-19T22:12:33.349254933Z "base_url":"http://solr-node2:8984/solr",
2026-05-19T22:12:33.349257016Z "collection":"test-collection",
2026-05-19T22:12:33.349259147Z "shard":"shard1",
2026-05-19T22:12:33.349261170Z "state":"down",
2026-05-19T22:12:33.349263156Z "type":"NRT",
2026-05-19T22:12:33.349265088Z "operation":"ADDREPLICA",
2026-05-19T22:12:33.349267150Z "waitForFinalState":"false"}
2026-05-19T22:12:33.457452923Z 2026-05-19 22:12:33.457 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-19T22:12:33.540219204Z 2026-05-19 22:12:33.539 INFO (qtp985324122-109-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-19T22:12:33.629798779Z 2026-05-19 22:12:33.629 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-19T22:12:33.690390903Z 2026-05-19 22:12:33.689 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-19T22:12:34.151572215Z 2026-05-19 22:12:34.151 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-19T22:12:34.151693348Z 2026-05-19 22:12:34.151 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-19T22:12:34.190371714Z 2026-05-19 22:12:34.189 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-19T22:12:35.453905708Z 2026-05-19 22:12:35.453 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-19T22:12:35.509244863Z 2026-05-19 22:12:35.508 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-19T22:12:35.518987157Z 2026-05-19 22:12:35.518 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-19T22:12:35.521358940Z 2026-05-19 22:12:35.520 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-19T22:12:35.521373642Z 2026-05-19 22:12:35.521 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-19T22:12:35.604772235Z 2026-05-19 22:12:35.604 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-19T22:12:35.647193179Z 2026-05-19 22:12:35.646 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-19T22:12:35.650102534Z 2026-05-19 22:12:35.649 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-19T22:12:35.652403563Z 2026-05-19 22:12:35.652 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-19T22:12:35.667120261Z 2026-05-19 22:12:35.666 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-19T22:12:35.675052460Z 2026-05-19 22:12:35.674 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-19T22:12:35.681145601Z 2026-05-19 22:12:35.680 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-19T22:12:35.681189251Z 2026-05-19 22:12:35.680 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-19T22:12:35.689456392Z 2026-05-19 22:12:35.689 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-19T22:12:35.695763371Z 2026-05-19 22:12:35.695 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-19T22:12:35.698485297Z 2026-05-19 22:12:35.698 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-19T22:12:35.711527868Z 2026-05-19 22:12:35.711 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-19T22:12:35.711567242Z 2026-05-19 22:12:35.711 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-19T22:12:35.711638583Z 2026-05-19 22:12:35.711 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-19T22:12:35.713473741Z 2026-05-19 22:12:35.713 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-19T22:12:35.714055056Z 2026-05-19 22:12:35.713 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-19T22:12:35.722091802Z 2026-05-19 22:12:35.721 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72083928546934784-core_node3-n_0000000000
2026-05-19T22:12:35.731659678Z 2026-05-19 22:12:35.731 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-19T22:12:35.836074794Z 2026-05-19 22:12:35.835 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 240] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-19T22:12:35.840169130Z 2026-05-19 22:12:35.839 INFO (qtp985324122-109-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-19T22:12:35.848947791Z 2026-05-19 22:12:35.848 INFO (qtp985324122-109-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2310
2026-05-19T22:12:35.858735565Z 2026-05-19 22:12:35.858 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-19T22:12:35.948653554Z 2026-05-19 22:12:35.948 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 246] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-19T22:12:35.951115590Z 2026-05-19 22:12:35.950 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2699
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1200844,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-19T22:12:20.448436377Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-19T22:12:26.149586451Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-19T22:12:26.149618137Z It should be set to 65000 to avoid operational disruption.
2026-05-19T22:12:26.149621114Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-19T22:12:26.149973846Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-19T22:12:26.149989473Z
2026-05-19T22:12:26.159656735Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-19T22:12:26.167533260Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-19T22:12:26.218720341Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-19T22:12:26.252778945Z WARNING: A command line option has enabled the Security Manager
2026-05-19T22:12:26.252796819Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-19T22:12:27.401321453Z 2026-05-19 22:12:27.397 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-19T22:12:27.676509006Z 2026-05-19 22:12:27.675 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-19T22:12:27.702346710Z 2026-05-19 22:12:27.701 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-19T22:12:27.710013392Z 2026-05-19 22:12:27.709 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-19T22:12:27.710034828Z 2026-05-19 22:12:27.709 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-19T22:12:27.710037708Z 2026-05-19 22:12:27.709 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-19T22:12:27.710050226Z 2026-05-19 22:12:27.709 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-19T22:12:27.709504888Z
2026-05-19T22:12:27.712960307Z 2026-05-19 22:12:27.712 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-19T22:12:27.735849388Z 2026-05-19 22:12:27.735 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-19T22:12:27.760147962Z 2026-05-19 22:12:27.759 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-19T22:12:27.900750413Z 2026-05-19 22:12:27.900 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-19T22:12:27.919192007Z 2026-05-19 22:12:27.918 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-19T22:12:27.919275063Z 2026-05-19 22:12:27.918 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-19T22:12:27.919398377Z 2026-05-19 22:12:27.918 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-19T22:12:28.029270090Z 2026-05-19 22:12:28.028 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-19T22:12:28.029620036Z 2026-05-19 22:12:28.029 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-19T22:12:28.029767883Z 2026-05-19 22:12:28.029 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-19T22:12:28.066228375Z 2026-05-19 22:12:28.065 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-19T22:12:28.805546061Z 2026-05-19 22:12:28.804 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-19T22:12:28.987939804Z 2026-05-19 22:12:28.986 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-19T22:12:28.989630883Z 2026-05-19 22:12:28.986 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-19T22:12:28.999131343Z 2026-05-19 22:12:28.998 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-19T22:12:29.007478745Z 2026-05-19 22:12:29.007 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-19T22:12:29.022767372Z 2026-05-19 22:12:29.022 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-19T22:12:29.022931942Z 2026-05-19 22:12:29.022 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-19T22:12:29.054228155Z 2026-05-19 22:12:29.053 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-19T22:12:29.063033403Z 2026-05-19 22:12:29.062 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-19T22:12:29.085509164Z 2026-05-19 22:12:29.084 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-19T22:12:29.098261407Z 2026-05-19 22:12:29.097 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-19T22:12:29.110123366Z 2026-05-19 22:12:29.109 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-19T22:12:29.121612223Z 2026-05-19 22:12:29.121 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-19T22:12:29.164764854Z 2026-05-19 22:12:29.164 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-19T22:12:29.282742160Z 2026-05-19 22:12:29.282 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-19T22:12:29.484396018Z 2026-05-19 22:12:29.483 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-19T22:12:29.683041148Z 2026-05-19 22:12:29.682 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-19T22:12:29.954326008Z 2026-05-19 22:12:29.953 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-19T22:12:30.106404492Z 2026-05-19 22:12:30.105 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-19T22:12:30.112513092Z 2026-05-19 22:12:30.112 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_19.request.log
2026-05-19T22:12:30.117038210Z 2026-05-19 22:12:30.116 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-19T22:12:30.118727652Z 2026-05-19 22:12:30.118 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3952ms
2026-05-19T22:12:30.292278958Z 2026-05-19 22:12:30.291 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=45
2026-05-19T22:12:30.358641821Z 2026-05-19 22:12:30.357 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-05-19T22:12:33.539339469Z 2026-05-19 22:12:33.538 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-19T22:12:33.600846183Z 2026-05-19 22:12:33.600 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-19T22:12:33.671787136Z 2026-05-19 22:12:33.671 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-19T22:12:33.978136343Z 2026-05-19 22:12:33.977 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-19T22:12:33.978160853Z 2026-05-19 22:12:33.977 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-19T22:12:33.996910999Z 2026-05-19 22:12:33.996 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-19T22:12:34.433932444Z 2026-05-19 22:12:34.433 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-19T22:12:34.453771654Z 2026-05-19 22:12:34.453 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-19T22:12:34.457289781Z 2026-05-19 22:12:34.456 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-19T22:12:34.458470036Z 2026-05-19 22:12:34.457 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-19T22:12:34.458484698Z 2026-05-19 22:12:34.458 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-19T22:12:34.501474342Z 2026-05-19 22:12:34.501 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-19T22:12:34.504499282Z 2026-05-19 22:12:34.504 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-19T22:12:34.506506555Z 2026-05-19 22:12:34.506 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-19T22:12:34.508359172Z 2026-05-19 22:12:34.508 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-19T22:12:34.520640891Z 2026-05-19 22:12:34.520 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-19T22:12:34.528660701Z 2026-05-19 22:12:34.528 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-19T22:12:34.536151011Z 2026-05-19 22:12:34.535 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-19T22:12:34.536170911Z 2026-05-19 22:12:34.535 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-19T22:12:34.542622238Z 2026-05-19 22:12:34.542 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-19T22:12:34.554274395Z 2026-05-19 22:12:34.553 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-19T22:12:34.555698378Z 2026-05-19 22:12:34.555 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-19T22:12:34.571558131Z 2026-05-19 22:12:34.571 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to
[...truncated too long message...]
pair and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Certificate was added to keystore
# Certificate was added to keystore
# Generating 3,072 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 90 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Certificate was added to keystore
# Certificate was added to keystore
# Certificate reply was installed in keystore
# Certificate was added to keystore
# Certificate was added to keystore
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/92-6
# Last output:
# INFO - 2026-05-19 22:13:38.120; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-05-19 22:13:38.123; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-05-19 22:13:38.134; org.apache.solr.common.util.EnvUtils; You are passing in deprecated system property host and should upgrade to using solr.host.advertise instead. The deprecated property support will be removed in future version of Solr.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is https://localhost:35101.
#
# ERROR: IOException occurred when talking to server at: https://localhost:35101/solr/admin/info/system
not ok 93 test keystore reload # in 2579 ms
# (in test file test/test_ssl.bats, line 529)
# `solr start --jvm-opts "-Dsolr.jetty.ssl.context.reload.scan.interval.secs=1 -DsocketTimeout=5000"' failed
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
# Generating 2,048 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 9,999 days
# for: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/93-7
# Last output:
# INFO - 2026-05-19 22:13:40.715; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO - 2026-05-19 22:13:40.718; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# WARN - 2026-05-19 22:13:40.730; org.apache.solr.common.util.EnvUtils; You are passing in deprecated system property host and should upgrade to using solr.host.advertise instead. The deprecated property support will be removed in future version of Solr.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is https://localhost:35101.
#
# ERROR: IOException occurred when talking to server at: https://localhost:35101/solr/admin/info/system
not ok 94 SOLR-11740 check 'solr stop' connection # in 224 ms
# (in test file test/test_start_solr.bats, line 32)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/94-1
not ok 95 stop command for single port # in 232 ms
# (in test file test/test_start_solr.bats, line 45)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/95-2
ok 96 check stop command doesn't hang # in 225 ms
not ok 97 SOLR-16976 solr starts with remote JMX enabled # in 233 ms
# (in test file test/test_start_solr.bats, line 77)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/97-4
not ok 98 deprecated system properties converted to modern properties # in 233 ms
# (in test file test/test_start_solr.bats, line 85)
# `solr start -Ddisable.config.edit=true' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #5 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/98-5
not ok 99 start with custom jetty options # in 227 ms
# (in test file test/test_start_solr.bats, line 93)
# `solr start --jettyconfig "--module=server"' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/99-6
not ok 100 bootstrapping a configset # in 233 ms
# (in test file test/test_start_solr.bats, line 104)
# `solr start -Dbootstrap_confdir="${confdir_path}" -Dcollection.configName=techproducts' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #7 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/100-7
not ok 101 status detects locally running solr # in 965 ms
# (in test file test/test_status.bats, line 34)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/101-1
# Last output:
#
# No Solr nodes are running.
not ok 102 status with --solr-url from user # in 222 ms
# (in test file test/test_status.bats, line 43)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/102-2
not ok 103 status with --port from user # in 227 ms
# (in test file test/test_status.bats, line 50)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/103-3
ok 104 multiple connection options are prevented # in 881 ms
ok 105 status with invalid --solr-url from user # in 1307 ms
not ok 106 status with --short format # in 234 ms
# (in test file test/test_status.bats, line 67)
# `solr start' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #6 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/106-6
not ok 107 setup_file failed
# (from function `setup_file' in test file test/test_stream.bats, line 22)
# `solr start -e techproducts' failed
#
# Starting up Solr on port 35101 using command:
# "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/bin/solr" start -p 35101 --solr-home "build/solr-10.1.0-SNAPSHOT/server/solr" --server-dir "/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server" -Dsolr.modules=clustering,extraction,langid,ltr,scripting -Dsolr.ltr.enabled=true -Dsolr.clustering.enabled=true
#
#
# ERROR: Port 35101 is already being used by another process.
#
ok 111 --version returns Solr version # in 747 ms
ok 112 version as direct tool call still runs # in 751 ms
not ok 113 setup_file failed
# (from function `setup_file' in test file test/test_zk.bats, line 22)
# `solr start -DminStateByteLenForCompression=0' failed
#
# Port 35101 is already being used by another process (pid: 1192605)
# Please choose a different port using the -p option.
#
ok 125 Cleanup # in 1887 ms
Build timed out (after 303 minutes). Marking the build as aborted.
Build timed out (after 303 minutes). Marking the build as failed.
Build was aborted
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 213 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/213/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-05-18 17:44:37.788; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 17:44:37.792; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 17:44:38.258; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 17:44:38.258; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1325 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 46655 is already being used by another process (pid: 1491086)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-18 17:44:39.463; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 17:44:39.467; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 17715 ms
ok 11 package with basic auth # in 12368 ms
ok 12 admin UI static content is excepted from basic auth # in 10737 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 145 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 78 ms
ok 16 compressed response can be decompressed and parsed # in 79 ms
ok 17 setting property # in 11208 ms
ok 18 short form of setting property # in 6561 ms
ok 19 connecting to solr via various solr urls and zk hosts # in 9308 ms
ok 20 create for non cloud mode # in 7879 ms
ok 21 create for cloud mode # in 9774 ms
ok 22 multiple connection options are prevented # in 799 ms
ok 23 create collection # in 7605 ms
ok 24 create collection using solrUrl # in 5264 ms
ok 25 create collection using legacy solrUrl # in 5406 ms
ok 26 create collection using Zookeeper # in 5252 ms
ok 27 reject d option with invalid config dir # in 2695 ms
ok 28 accept d option with builtin config # in 5121 ms
ok 29 accept d option with explicit path to config # in 5245 ms
ok 30 accept n option as config name # in 5159 ms
ok 31 allow config reuse when n option specifies same config # in 8983 ms
ok 32 create multisharded collections when s provided # in 5078 ms
ok 33 create replicated collections when rf provided # in 6050 ms
ok 34 can delete collections # in 10574 ms
ok 35 can delete collections with solr-url # in 8321 ms
ok 36 collection delete also deletes zk config # in 7234 ms
ok 37 deletes accompanying zk config with nondefault name # in 7243 ms
ok 38 delete-config option can opt to leave config in zk # in 7023 ms
ok 39 start -e cloud works with --jvm-opts # in 23929 ms
ok 40 start -e cloud works with --no-prompt # in 46856 ms
ok 41 Check export command # in 19496 ms
ok 42 export fails on non cloud mode # in 9775 ms
Waiting for Tika Server to be ready on port 46660
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 43 using curl to extract a single pdf file # in 16639 ms
WARNING: Docker not available; bypassing test.
ok 44 using the bin/solr post tool to extract content from pdf # in 16569 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to crawl web site # in 16585 ms
Stopping Tika Server container
ok 46 healthcheck on cloud solr # in 10581 ms
ok 47 healthcheck errors on standalone solr # in 11288 ms
ok 48 solr help flag prints help # in 739 ms
ok 49 solr with no flags prints help # in 745 ms
ok 50 start help flag prints help # in 146 ms
ok 51 start h flag prints help # in 147 ms
ok 52 stop help flag prints help # in 144 ms
ok 53 restart help flag prints help # in 147 ms
ok 54 status help flag prints help # in 794 ms
ok 55 healthcheck help flag prints help # in 754 ms
ok 56 create help flag prints help # in 769 ms
ok 57 delete help flag prints help # in 760 ms
ok 58 zk help flag prints help # in 785 ms
ok 59 auth help flag prints help # in 761 ms
ok 60 assert help flag prints help # in 761 ms
ok 61 post help flag prints help # in 762 ms
ok 62 SQL Module # in 15730 ms
ok 63 icu collation in analysis-extras module # in 12577 ms
ok 64 Check lifecycle of sentiment classification # in 23712 ms
ok 65 lifecycle of package # in 8842 ms
ok 66 deploying and undeploying a collection level package # in 13660 ms
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 68 Affinity placement plugin using sysprop # in 15332 ms
ok 69 Random placement plugin using ENV # in 15351 ms
ok 70 Check help command # in 2445 ms
ok 71 basic post with a type specified # in 5431 ms
ok 72 basic post with collection instead of url parameter # in 4070 ms
ok 73 basic post with solr-url and collection # in 4054 ms
ok 74 basic post WITHOUT a type specified # in 11569 ms
ok 75 crawling a directory as a dry-run # in 792 ms
ok 76 crawling a directory # in 4454 ms
ok 77 skipcommit and optimize and delete # in 3890 ms
ok 78 args mode # in 6917 ms
ok 79 stdin mode # in 3512 ms
ok 80 verbose echo the Solr response # in 3855 ms
ok 81 post solr log into solr via cli # in 9354 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 16 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-18T17:54:53.686231180Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-18T17:54:53.686268053Z It should be set to 65000 to avoid operational disruption.
2026-05-18T17:54:53.686271123Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-18T17:54:53.686552110Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-18T17:54:53.686564062Z
2026-05-18T17:54:53.696322220Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-18T17:54:53.703857425Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-18T17:54:53.748370063Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-18T17:54:53.782413146Z WARNING: A command line option has enabled the Security Manager
2026-05-18T17:54:53.782439175Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-18T17:54:54.881051302Z 2026-05-18 17:54:54.877 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-18T17:54:55.149476988Z 2026-05-18 17:54:55.148 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-18T17:54:55.174271480Z 2026-05-18 17:54:55.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-18T17:54:55.181097296Z 2026-05-18 17:54:55.180 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-18T17:54:55.181150739Z 2026-05-18 17:54:55.180 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-18T17:54:55.181296354Z 2026-05-18 17:54:55.180 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-18T17:54:55.181393996Z 2026-05-18 17:54:55.180 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-18T17:54:55.180890227Z
2026-05-18T17:54:55.183856000Z 2026-05-18 17:54:55.183 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-18T17:54:55.204698339Z 2026-05-18 17:54:55.204 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-18T17:54:55.217514135Z 2026-05-18 17:54:55.217 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-18T17:54:55.217615905Z 2026-05-18 17:54:55.217 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-18T17:54:55.251169392Z 2026-05-18 17:54:55.250 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-18T17:54:55.970084748Z 2026-05-18 17:54:55.969 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-18T17:54:56.211204373Z 2026-05-18 17:54:56.210 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-18T17:54:56.216123931Z 2026-05-18 17:54:56.215 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-18T17:54:56.216174087Z 2026-05-18 17:54:56.215 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-18T17:54:56.218568815Z 2026-05-18 17:54:56.218 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-18T17:54:56.218648932Z 2026-05-18 17:54:56.218 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-18T17:54:56.254239808Z 2026-05-18 17:54:56.253 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-18T17:54:56.744245252Z 2026-05-18 17:54:56.742 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-18T17:54:56.761366154Z 2026-05-18 17:54:56.760 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T17:54:56.808236732Z 2026-05-18 17:54:56.807 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-18T17:54:56.854070099Z 2026-05-18 17:54:56.853 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T17:54:56.854137446Z 2026-05-18 17:54:56.853 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T17:54:56.991117997Z 2026-05-18 17:54:56.990 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-18T17:54:57.019969053Z 2026-05-18 17:54:57.019 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T17:54:57.037341974Z 2026-05-18 17:54:57.036 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-18T17:54:57.043767485Z 2026-05-18 17:54:57.043 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72077321711386624-solr-node1:8983_solr-n_0000000000) starting
2026-05-18T17:54:57.113514654Z 2026-05-18 17:54:57.112 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-18T17:54:57.114006597Z 2026-05-18 17:54:57.113 INFO (OverseerStateUpdate-72077321711386624-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-18T17:54:57.127536057Z 2026-05-18 17:54:57.127 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-18T17:54:57.171319104Z 2026-05-18 17:54:57.170 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-18T17:54:57.558244263Z 2026-05-18 17:54:57.556 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-18T17:54:57.755461485Z 2026-05-18 17:54:57.754 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-18T17:54:58.031023488Z 2026-05-18 17:54:58.030 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T17:54:58.230448553Z 2026-05-18 17:54:58.229 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-18T17:54:58.310541276Z 2026-05-18 17:54:58.310 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_18.request.log
2026-05-18T17:54:58.314896830Z 2026-05-18 17:54:58.314 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-18T17:54:58.315719700Z 2026-05-18 17:54:58.315 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4613ms
2026-05-18T17:54:58.537260797Z 2026-05-18 17:54:58.536 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=51
2026-05-18T17:54:58.604081089Z 2026-05-18 17:54:58.603 INFO (qtp985324122-97-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-05-18T17:55:02.205195073Z 2026-05-18 17:55:02.204 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-18T17:55:02.341371875Z 2026-05-18 17:55:02.340 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-18T17:55:06.378794292Z 2026-05-18 17:55:06.377 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-05-18T17:55:06.406475114Z 2026-05-18 17:55:06.406 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-05-18T17:55:06.428603085Z 2026-05-18 17:55:06.428 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-18T17:55:07.034970450Z 2026-05-18 17:55:07.034 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-18T17:55:07.052261281Z 2026-05-18 17:55:07.051 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-18T17:55:07.057152304Z 2026-05-18 17:55:07.056 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:07.096433474Z 2026-05-18 17:55:07.095 INFO (OverseerStateUpdate-72077321711386624-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-18T17:55:07.096454887Z "core":"test-collection_shard3_replica_n1",
2026-05-18T17:55:07.096457601Z "node_name":"solr-node1:8983_solr",
2026-05-18T17:55:07.096459986Z "base_url":"http://solr-node1:8983/solr",
2026-05-18T17:55:07.096462534Z "collection":"test-collection",
2026-05-18T17:55:07.096464564Z "shard":"shard3",
2026-05-18T17:55:07.096466652Z "state":"down",
2026-05-18T17:55:07.096468646Z "type":"NRT",
2026-05-18T17:55:07.096470626Z "operation":"ADDREPLICA",
2026-05-18T17:55:07.096472643Z "waitForFinalState":"false"}
2026-05-18T17:55:07.123779621Z 2026-05-18 17:55:07.123 INFO (OverseerStateUpdate-72077321711386624-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-18T17:55:07.123801645Z "core":"test-collection_shard2_replica_n2",
2026-05-18T17:55:07.123804355Z "node_name":"solr-node3:8985_solr",
2026-05-18T17:55:07.123806744Z "base_url":"http://solr-node3:8985/solr",
2026-05-18T17:55:07.123808870Z "collection":"test-collection",
2026-05-18T17:55:07.123811075Z "shard":"shard2",
2026-05-18T17:55:07.123813046Z "state":"down",
2026-05-18T17:55:07.123814998Z "type":"NRT",
2026-05-18T17:55:07.123816909Z "operation":"ADDREPLICA",
2026-05-18T17:55:07.123818867Z "waitForFinalState":"false"}
2026-05-18T17:55:07.134052260Z 2026-05-18 17:55:07.133 INFO (OverseerStateUpdate-72077321711386624-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-18T17:55:07.134075800Z "core":"test-collection_shard1_replica_n4",
2026-05-18T17:55:07.134079918Z "node_name":"solr-node2:8984_solr",
2026-05-18T17:55:07.134095472Z "base_url":"http://solr-node2:8984/solr",
2026-05-18T17:55:07.134098556Z "collection":"test-collection",
2026-05-18T17:55:07.134100756Z "shard":"shard1",
2026-05-18T17:55:07.134102727Z "state":"down",
2026-05-18T17:55:07.134104630Z "type":"NRT",
2026-05-18T17:55:07.134106673Z "operation":"ADDREPLICA",
2026-05-18T17:55:07.134108654Z "waitForFinalState":"false"}
2026-05-18T17:55:07.255470862Z 2026-05-18 17:55:07.254 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:07.333274958Z 2026-05-18 17:55:07.332 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-18T17:55:07.451483591Z 2026-05-18 17:55:07.451 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-18T17:55:07.539721695Z 2026-05-18 17:55:07.539 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-18T17:55:07.952882258Z 2026-05-18 17:55:07.952 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-18T17:55:07.952984570Z 2026-05-18 17:55:07.952 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-18T17:55:07.999083497Z 2026-05-18 17:55:07.998 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-18T17:55:08.864815124Z 2026-05-18 17:55:08.863 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T17:55:08.890159587Z 2026-05-18 17:55:08.889 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-18T17:55:08.893989232Z 2026-05-18 17:55:08.893 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-18T17:55:08.894757808Z 2026-05-18 17:55:08.894 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-18T17:55:08.894773442Z 2026-05-18 17:55:08.894 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-18T17:55:08.990866246Z 2026-05-18 17:55:08.989 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-18T17:55:08.995997559Z 2026-05-18 17:55:08.995 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-18T17:55:08.998115687Z 2026-05-18 17:55:08.997 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-18T17:55:09.008466588Z 2026-05-18 17:55:09.008 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-18T17:55:09.074497675Z 2026-05-18 17:55:09.073 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-18T17:55:09.083872190Z 2026-05-18 17:55:09.083 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-18T17:55:09.091482147Z 2026-05-18 17:55:09.091 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-18T17:55:09.091502814Z 2026-05-18 17:55:09.091 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-18T17:55:09.102962362Z 2026-05-18 17:55:09.102 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-18T17:55:09.114470976Z 2026-05-18 17:55:09.108 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-18T17:55:09.116162180Z 2026-05-18 17:55:09.115 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-18T17:55:09.129390148Z 2026-05-18 17:55:09.129 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-18T17:55:09.129408210Z 2026-05-18 17:55:09.129 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-18T17:55:09.129449080Z 2026-05-18 17:55:09.129 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-18T17:55:09.130718771Z 2026-05-18 17:55:09.130 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-18T17:55:09.131263539Z 2026-05-18 17:55:09.131 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-18T17:55:09.132329602Z 2026-05-18 17:55:09.132 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72077321711386624-core_node3-n_0000000000
2026-05-18T17:55:09.141171208Z 2026-05-18 17:55:09.140 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-18T17:55:09.248031757Z 2026-05-18 17:55:09.247 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:09.251495757Z 2026-05-18 17:55:09.251 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-18T17:55:09.259886925Z 2026-05-18 17:55:09.259 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1930
2026-05-18T17:55:09.360239712Z 2026-05-18 17:55:09.359 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1523611,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-18T17:54:53.441260974Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-18T17:54:58.929867108Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-18T17:54:58.929902702Z It should be set to 65000 to avoid operational disruption.
2026-05-18T17:54:58.929905759Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-18T17:54:58.930191646Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-18T17:54:58.930201938Z
2026-05-18T17:54:58.939747916Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-18T17:54:58.948651155Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-18T17:54:58.993830494Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-18T17:54:59.028273039Z WARNING: A command line option has enabled the Security Manager
2026-05-18T17:54:59.028305374Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-18T17:55:00.165091264Z 2026-05-18 17:55:00.161 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-18T17:55:00.440605034Z 2026-05-18 17:55:00.440 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-18T17:55:00.466814133Z 2026-05-18 17:55:00.466 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-18T17:55:00.474577085Z 2026-05-18 17:55:00.473 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-18T17:55:00.474709164Z 2026-05-18 17:55:00.474 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-18T17:55:00.474866447Z 2026-05-18 17:55:00.474 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-18T17:55:00.475031642Z 2026-05-18 17:55:00.474 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-18T17:55:00.474210137Z
2026-05-18T17:55:00.477997472Z 2026-05-18 17:55:00.477 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-18T17:55:00.501692610Z 2026-05-18 17:55:00.501 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-18T17:55:00.525860489Z 2026-05-18 17:55:00.525 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T17:55:00.673526853Z 2026-05-18 17:55:00.672 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-18T17:55:00.696689159Z 2026-05-18 17:55:00.696 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T17:55:00.696794271Z 2026-05-18 17:55:00.696 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T17:55:00.696916380Z 2026-05-18 17:55:00.696 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T17:55:00.807748927Z 2026-05-18 17:55:00.807 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-18T17:55:00.807982972Z 2026-05-18 17:55:00.807 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-18T17:55:00.808066733Z 2026-05-18 17:55:00.807 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-18T17:55:00.844633814Z 2026-05-18 17:55:00.843 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-18T17:55:01.839202236Z 2026-05-18 17:55:01.838 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-18T17:55:02.073865990Z 2026-05-18 17:55:02.073 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-18T17:55:02.076017125Z 2026-05-18 17:55:02.073 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-18T17:55:02.087053651Z 2026-05-18 17:55:02.086 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T17:55:02.092931604Z 2026-05-18 17:55:02.092 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-18T17:55:02.096944102Z 2026-05-18 17:55:02.096 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T17:55:02.096958320Z 2026-05-18 17:55:02.096 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T17:55:02.133648931Z 2026-05-18 17:55:02.133 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-18T17:55:02.142841675Z 2026-05-18 17:55:02.142 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-18T17:55:02.167813665Z 2026-05-18 17:55:02.167 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T17:55:02.182197454Z 2026-05-18 17:55:02.181 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-18T17:55:02.194771674Z 2026-05-18 17:55:02.194 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-18T17:55:02.208545400Z 2026-05-18 17:55:02.208 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-18T17:55:02.255649797Z 2026-05-18 17:55:02.255 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-18T17:55:02.340778399Z 2026-05-18 17:55:02.340 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-18T17:55:02.534789449Z 2026-05-18 17:55:02.534 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-18T17:55:02.724979929Z 2026-05-18 17:55:02.724 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-18T17:55:03.006618759Z 2026-05-18 17:55:03.006 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T17:55:03.178752878Z 2026-05-18 17:55:03.177 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-18T17:55:03.186574440Z 2026-05-18 17:55:03.186 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_18.request.log
2026-05-18T17:55:03.191038891Z 2026-05-18 17:55:03.190 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-18T17:55:03.193812224Z 2026-05-18 17:55:03.192 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4245ms
2026-05-18T17:55:03.415410459Z 2026-05-18 17:55:03.414 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=80
2026-05-18T17:55:03.470594967Z 2026-05-18 17:55:03.470 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-05-18T17:55:07.332272081Z 2026-05-18 17:55:07.331 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-18T17:55:07.388533494Z 2026-05-18 17:55:07.387 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-18T17:55:07.449137314Z 2026-05-18 17:55:07.448 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-18T17:55:07.728321097Z 2026-05-18 17:55:07.727 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-18T17:55:07.728350295Z 2026-05-18 17:55:07.727 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-18T17:55:07.748379827Z 2026-05-18 17:55:07.747 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-18T17:55:08.195958761Z 2026-05-18 17:55:08.195 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T17:55:08.214067524Z 2026-05-18 17:55:08.213 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-18T17:55:08.217500722Z 2026-05-18 17:55:08.217 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-18T17:55:08.218589353Z 2026-05-18 17:55:08.217 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-18T17:55:08.218689072Z 2026-05-18 17:55:08.218 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-18T17:55:08.347723784Z 2026-05-18 17:55:08.347 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-18T17:55:08.350869728Z 2026-05-18 17:55:08.350 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-18T17:55:08.352805901Z 2026-05-18 17:55:08.352 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-18T17:55:08.358080671Z 2026-05-18 17:55:08.357 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-18T17:55:08.370918380Z 2026-05-18 17:55:08.370 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-18T17:55:08.379171434Z 2026-05-18 17:55:08.378 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-18T17:55:08.385337104Z 2026-05-18 17:55:08.384 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-18T17:55:08.385358826Z 2026-05-18 17:55:08.385 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-18T17:55:08.392417284Z 2026-05-18 17:55:08.392 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-18T17:55:08.403700644Z 2026-05-18 17:55:08.403 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-18T17:55:08.405138366Z 2026-05-18 17:55:08.404 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-18T17:55:08.418572930Z 2026-05-18 17:55:08.418 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-18T17:55:08.418591094Z 2026-05-18 17:55:08.418 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-18T17:55:08.418630189Z 2026-05-18 17:55:08.418 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-18T17:55:08.419902953Z 2026-05-18 17:55:08.419 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-18T17:55:08.420428074Z 2026-05-18 17:55:08.420 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-18T17:55:08.421520075Z 2026-05-18 17:55:08.421 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72077321711386627-core_node6-n_0000000000
2026-05-18T17:55:08.430352900Z 2026-05-18 17:55:08.430 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-18T17:55:08.660118193Z 2026-05-18 17:55:08.659 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:08.671922012Z 2026-05-18 17:55:08.671 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-18T17:55:08.681599879Z 2026-05-18 17:55:08.681 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1350
2026-05-18T17:55:08.785791241Z 2026-05-18 17:55:08.785 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:09.248687339Z 2026-05-18 17:55:09.247 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:09.360481388Z 2026-05-18 17:55:09.360 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1524007,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-18T17:54:58.696092154Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-18T17:54:59.166797714Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-18T17:54:59.166835540Z It should be set to 65000 to avoid operational disruption.
2026-05-18T17:54:59.166838783Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-18T17:54:59.166840983Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-18T17:54:59.166843039Z
2026-05-18T17:54:59.176043753Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-18T17:54:59.183584949Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-18T17:54:59.229451439Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-18T17:54:59.260472131Z WARNING: A command line option has enabled the Security Manager
2026-05-18T17:54:59.260498568Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-18T17:55:00.514307514Z 2026-05-18 17:55:00.510 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-18T17:55:00.789848095Z 2026-05-18 17:55:00.789 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-18T17:55:00.818102900Z 2026-05-18 17:55:00.817 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-18T17:55:00.826287890Z 2026-05-18 17:55:00.825 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-18T17:55:00.826318218Z 2026-05-18 17:55:00.825 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-18T17:55:00.826485432Z 2026-05-18 17:55:00.825 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-18T17:55:00.826696919Z 2026-05-18 17:55:00.825 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-18T17:55:00.825833707Z
2026-05-18T17:55:00.829242963Z 2026-05-18 17:55:00.828 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-18T17:55:00.853005733Z 2026-05-18 17:55:00.852 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-18T17:55:00.881766925Z 2026-05-18 17:55:00.881 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T17:55:01.026128490Z 2026-05-18 17:55:01.025 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-18T17:55:01.045259261Z 2026-05-18 17:55:01.044 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T17:55:01.045408507Z 2026-05-18 17:55:01.044 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T17:55:01.045552339Z 2026-05-18 17:55:01.044 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T17:55:01.155496627Z 2026-05-18 17:55:01.154 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-18T17:55:01.155905292Z 2026-05-18 17:55:01.155 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-18T17:55:01.156036272Z 2026-05-18 17:55:01.155 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-18T17:55:01.194578953Z 2026-05-18 17:55:01.193 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-18T17:55:02.059298521Z 2026-05-18 17:55:02.058 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-18T17:55:02.208974260Z 2026-05-18 17:55:02.208 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-18T17:55:02.210447117Z 2026-05-18 17:55:02.208 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-18T17:55:02.219574345Z 2026-05-18 17:55:02.219 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T17:55:02.224503810Z 2026-05-18 17:55:02.224 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-18T17:55:02.228081261Z 2026-05-18 17:55:02.227 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T17:55:02.228165714Z 2026-05-18 17:55:02.227 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T17:55:02.274929428Z 2026-05-18 17:55:02.274 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-18T17:55:02.283245324Z 2026-05-18 17:55:02.282 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-18T17:55:02.309065104Z 2026-05-18 17:55:02.308 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T17:55:02.320814848Z 2026-05-18 17:55:02.320 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-18T17:55:02.332017933Z 2026-05-18 17:55:02.331 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-18T17:55:02.342359391Z 2026-05-18 17:55:02.341 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-18T17:55:02.386020214Z 2026-05-18 17:55:02.385 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-18T17:55:02.658646622Z 2026-05-18 17:55:02.658 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-18T17:55:02.841147285Z 2026-05-18 17:55:02.840 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-18T17:55:03.100850947Z 2026-05-18 17:55:03.100 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T17:55:03.238453855Z 2026-05-18 17:55:03.237 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-18T17:55:03.244777135Z 2026-05-18 17:55:03.244 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_18.request.log
2026-05-18T17:55:03.248836444Z 2026-05-18 17:55:03.248 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-18T17:55:03.249725051Z 2026-05-18 17:55:03.249 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4067ms
2026-05-18T17:55:05.210914081Z 2026-05-18 17:55:05.203 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=71
2026-05-18T17:55:05.281130200Z 2026-05-18 17:55:05.280 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=21
2026-05-18T17:55:07.002525708Z 2026-05-18 17:55:07.001 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=11
2026-05-18T17:55:07.333966271Z 2026-05-18 17:55:07.333 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-18T17:55:07.437018456Z 2026-05-18 17:55:07.436 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-18T17:55:07.508169025Z 2026-05-18 17:55:07.507 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-18T17:55:07.824947566Z 2026-05-18 17:55:07.824 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-18T17:55:07.825013791Z 2026-05-18 17:55:07.824 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-18T17:55:07.844378910Z 2026-05-18 17:55:07.843 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-18T17:55:08.307582600Z 2026-05-18 17:55:08.307 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T17:55:08.330406375Z 2026-05-18 17:55:08.329 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-18T17:55:08.338781013Z 2026-05-18 17:55:08.338 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-18T17:55:08.340149988Z 2026-05-18 17:55:08.339 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-18T17:55:08.340205396Z 2026-05-18 17:55:08.339 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-18T17:55:08.390636916Z 2026-05-18 17:55:08.390 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-18T17:55:08.393807635Z 2026-05-18 17:55:08.393 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-18T17:55:08.395645664Z 2026-05-18 17:55:08.395 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-18T17:55:08.398210937Z 2026-05-18 17:55:08.397 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-18T17:55:08.410731683Z 2026-05-18 17:55:08.410 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-18T17:55:08.419124009Z 2026-05-18 17:55:08.418 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-18T17:55:08.425980796Z 2026-05-18 17:55:08.425 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-18T17:55:08.426019363Z 2026-05-18 17:55:08.425 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-18T17:55:08.432156347Z 2026-05-18 17:55:08.431 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-18T17:55:08.441432526Z 2026-05-18 17:55:08.440 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-18T17:55:08.443035910Z 2026-05-18 17:55:08.442 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-18T17:55:08.510132545Z 2026-05-18 17:55:08.509 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-18T17:55:08.510186194Z 2026-05-18 17:55:08.509 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-18T17:55:08.510190663Z 2026-05-18 17:55:08.509 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-18T17:55:08.512478153Z 2026-05-18 17:55:08.512 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-18T17:55:08.513282771Z 2026-05-18 17:55:08.512 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-18T17:55:08.514805824Z 2026-05-18 17:55:08.514 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72077321711386628-core_node5-n_0000000000
2026-05-18T17:55:08.530894674Z 2026-05-18 17:55:08.530 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-18T17:55:08.660117198Z 2026-05-18 17:55:08.659 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:08.671889710Z 2026-05-18 17:55:08.671 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-18T17:55:08.682514737Z 2026-05-18 17:55:08.682 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1350
2026-05-18T17:55:08.785370234Z 2026-05-18 17:55:08.784 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:09.248721995Z 2026-05-18 17:55:09.247 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:09.274080687Z 2026-05-18 17:55:09.273 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-18T17:55:09.360582411Z 2026-05-18 17:55:09.360 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T17:55:09.363506702Z 2026-05-18 17:55:09.363 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2348
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1524190,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-18T17:54:58.92905747Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 82 Docker SolrCloud rolling upgrade # in 27353 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Download complete
# c937aee9a870: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Already exists
# 9d5a0b6cb36c: Already exists
# 49518c28c0b3: Already exists
# 41162acdcc29: Already exists
# 2ed0fd558ca3: Already exists
# 8b0357ffbe54: Pulling fs layer
# 0df26b0d5a69: Pulling fs layer
# a26fc87814fe: Pulling fs layer
# 67860c4a05fe: Pulling fs layer
# d1a380e5a7b1: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 67860c4a05fe: Waiting
# d1a380e5a7b1: Waiting
# 4f4fb700ef54: Waiting
# 0df26b0d5a69: Verifying Checksum
# 0df26b0d5a69: Download complete
# a26fc87814fe: Verifying Checksum
# a26fc87814fe: Download complete
# 67860c4a05fe: Verifying Checksum
# 67860c4a05fe: Download complete
# d1a380e5a7b1: Verifying Checksum
# d1a380e5a7b1: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 8b0357ffbe54: Verifying Checksum
# 8b0357ffbe54: Download complete
# 8b0357ffbe54: Pull complete
# 0df26b0d5a69: Pull complete
# a26fc87814fe: Pull complete
# 67860c4a05fe: Pull complete
# d1a380e5a7b1: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:16825c9cb8e05e7a6dd9d4209615781d30bc3137a6017a5254cec93560c7c8ba
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# e14dc15402a2f8778175fbfc836bdcca228d09aae753cd0b8d83afdfbd5c8776
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 2d2f713ab0644af8419216c384cf9079af849dc1ede9dda2a47bf36434b01e2a
# Starting solr-node2 connected to first node's ZooKeeper
# b7839594e47ac596038e4430a76278579355b7bef12b9313bf38d7d903e10f50
# Started solr-node3 connected to first node's ZooKeeper
# bfcdd01f992baea4dab026706e43fee4c6a36f34f570fb3ef4d61cbb097796de
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-18 17:55:06.470; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 17:55:06.506; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 83 allowPaths - backup # in 15529 ms
ok 84 snapshot lifecycle # in 4866 ms
ok 85 snapshot list # in 3490 ms
ok 86 snapshot describe # in 3410 ms
ok 87 start solr with ssl # in 16704 ms
ok 88 use different hostname when not checking peer-name # in 25173 ms
ok 89 start solr with ssl and auth # in 14789 ms
ok 90 start solr with client truststore and security manager # in 17328 ms
ok 91 start solr with mTLS needed # in 48729 ms
ok 92 start solr with mTLS wanted # in 35751 ms
ok 93 test keystore reload # in 61815 ms
ok 94 SOLR-11740 check 'solr stop' connection # in 16097 ms
ok 95 stop command for single port # in 18820 ms
ok 96 check stop command doesn't hang # in 236 ms
ok 97 SOLR-16976 solr starts with remote JMX enabled # in 8790 ms
ok 98 deprecated system properties converted to modern properties # in 195243 ms
ok 99 start with custom jetty options # in 8429 ms
ok 100 bootstrapping a configset # in 9813 ms
ok 101 status detects locally running solr # in 10131 ms
ok 102 status with --solr-url from user # in 8405 ms
ok 103 status with --port from user # in 8280 ms
ok 104 multiple connection options are prevented # in 910 ms
ok 105 status with invalid --solr-url from user # in 1316 ms
ok 106 status with --short format # in 196063 ms
ok 107 searching solr via locally executed streaming expression # in 1717 ms
ok 108 searching solr via remotely executed streaming expression # in 1254 ms
ok 109 variable interpolation # in 1659 ms
ok 110 searching solr without credentials fails with error # in 1182 ms
ok 111 --version returns Solr version # in 754 ms
ok 112 version as direct tool call still runs # in 748 ms
ok 113 short help # in 845 ms
ok 114 short help is inferred # in 817 ms
ok 115 long help # in 809 ms
ok 116 running subcommands with zk is prevented # in 750 ms
ok 117 listing out files # in 2208 ms
ok 118 connecting to solr via various solr urls and zk hosts # in 8053 ms
ok 119 copying files around # in 11104 ms
ok 120 upconfig # in 2646 ms
ok 121 SOLR-12429 test upconfig fails with symlink # in 1131 ms
ok 122 downconfig # in 1195 ms
ok 123 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1428 ms
ok 124 env var ZK_HOST is honored # in 3204 ms
ok 125 Cleanup # in 1885 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 41s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vagfyxc4bxbn6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 212 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/212/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-05-18 06:16:54.507; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 06:16:54.511; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 06:16:54.975; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 06:16:54.975; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1295 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 41823 is already being used by another process (pid: 1702748)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-18 06:16:56.140; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 06:16:56.143; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 17731 ms
ok 11 package with basic auth # in 12023 ms
ok 12 admin UI static content is excepted from basic auth # in 10653 ms
ok 13 nothing # in 9 ms
ok 14 server does not compress response without Accept-Encoding header # in 145 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 79 ms
ok 16 compressed response can be decompressed and parsed # in 78 ms
ok 17 setting property # in 11019 ms
ok 18 short form of setting property # in 6532 ms
ok 19 connecting to solr via various solr urls and zk hosts # in 9305 ms
ok 20 create for non cloud mode # in 7688 ms
ok 21 create for cloud mode # in 9902 ms
ok 22 multiple connection options are prevented # in 830 ms
ok 23 create collection # in 7770 ms
ok 24 create collection using solrUrl # in 5372 ms
ok 25 create collection using legacy solrUrl # in 5198 ms
ok 26 create collection using Zookeeper # in 5264 ms
ok 27 reject d option with invalid config dir # in 2616 ms
ok 28 accept d option with builtin config # in 5132 ms
ok 29 accept d option with explicit path to config # in 5200 ms
ok 30 accept n option as config name # in 5161 ms
ok 31 allow config reuse when n option specifies same config # in 8958 ms
ok 32 create multisharded collections when s provided # in 5128 ms
ok 33 create replicated collections when rf provided # in 6113 ms
ok 34 can delete collections # in 10786 ms
ok 35 can delete collections with solr-url # in 8548 ms
ok 36 collection delete also deletes zk config # in 7524 ms
ok 37 deletes accompanying zk config with nondefault name # in 7317 ms
ok 38 delete-config option can opt to leave config in zk # in 7098 ms
ok 39 start -e cloud works with --jvm-opts # in 21953 ms
ok 40 start -e cloud works with --no-prompt # in 18483 ms
ok 41 Check export command # in 18774 ms
ok 42 export fails on non cloud mode # in 9326 ms
Waiting for Tika Server to be ready on port 41828
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 43 using curl to extract a single pdf file # in 16822 ms
WARNING: Docker not available; bypassing test.
ok 44 using the bin/solr post tool to extract content from pdf # in 16568 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to crawl web site # in 16558 ms
Stopping Tika Server container
ok 46 healthcheck on cloud solr # in 10087 ms
ok 47 healthcheck errors on standalone solr # in 11208 ms
ok 48 solr help flag prints help # in 736 ms
ok 49 solr with no flags prints help # in 757 ms
ok 50 start help flag prints help # in 147 ms
ok 51 start h flag prints help # in 149 ms
ok 52 stop help flag prints help # in 143 ms
ok 53 restart help flag prints help # in 145 ms
ok 54 status help flag prints help # in 825 ms
ok 55 healthcheck help flag prints help # in 747 ms
ok 56 create help flag prints help # in 775 ms
ok 57 delete help flag prints help # in 748 ms
ok 58 zk help flag prints help # in 744 ms
ok 59 auth help flag prints help # in 759 ms
ok 60 assert help flag prints help # in 757 ms
ok 61 post help flag prints help # in 761 ms
ok 62 SQL Module # in 15465 ms
ok 63 icu collation in analysis-extras module # in 12179 ms
ok 64 Check lifecycle of sentiment classification # in 23948 ms
ok 65 lifecycle of package # in 8485 ms
ok 66 deploying and undeploying a collection level package # in 13186 ms
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 68 Affinity placement plugin using sysprop # in 15379 ms
ok 69 Random placement plugin using ENV # in 15941 ms
ok 70 Check help command # in 2333 ms
ok 71 basic post with a type specified # in 5506 ms
ok 72 basic post with collection instead of url parameter # in 4014 ms
ok 73 basic post with solr-url and collection # in 3987 ms
ok 74 basic post WITHOUT a type specified # in 11770 ms
ok 75 crawling a directory as a dry-run # in 823 ms
ok 76 crawling a directory # in 4411 ms
ok 77 skipcommit and optimize and delete # in 3867 ms
ok 78 args mode # in 7397 ms
ok 79 stdin mode # in 3621 ms
ok 80 verbose echo the Solr response # in 3874 ms
ok 81 post solr log into solr via cli # in 9358 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-18T06:26:39.982629565Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-18T06:26:39.982666502Z It should be set to 65000 to avoid operational disruption.
2026-05-18T06:26:39.982669757Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-18T06:26:39.982948127Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-18T06:26:39.982964632Z
2026-05-18T06:26:39.992020522Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-18T06:26:39.999912970Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-18T06:26:40.041246096Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-18T06:26:40.073174139Z WARNING: A command line option has enabled the Security Manager
2026-05-18T06:26:40.073193924Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-18T06:26:41.226955731Z 2026-05-18 06:26:41.223 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-18T06:26:41.492892271Z 2026-05-18 06:26:41.492 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-18T06:26:41.517416489Z 2026-05-18 06:26:41.516 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-18T06:26:41.524642572Z 2026-05-18 06:26:41.524 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-18T06:26:41.524735767Z 2026-05-18 06:26:41.524 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-18T06:26:41.524833683Z 2026-05-18 06:26:41.524 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-18T06:26:41.524966512Z 2026-05-18 06:26:41.524 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-18T06:26:41.524391144Z
2026-05-18T06:26:41.527557875Z 2026-05-18 06:26:41.527 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-18T06:26:41.549805793Z 2026-05-18 06:26:41.549 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-18T06:26:41.563084287Z 2026-05-18 06:26:41.562 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-18T06:26:41.563151518Z 2026-05-18 06:26:41.562 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-18T06:26:41.599526146Z 2026-05-18 06:26:41.598 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-18T06:26:42.283185356Z 2026-05-18 06:26:42.282 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-18T06:26:42.955247887Z 2026-05-18 06:26:42.919 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-18T06:26:42.957058906Z 2026-05-18 06:26:42.949 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-18T06:26:42.957155409Z 2026-05-18 06:26:42.949 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-18T06:26:42.961676620Z 2026-05-18 06:26:42.961 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-18T06:26:42.961773272Z 2026-05-18 06:26:42.961 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-18T06:26:43.327565741Z 2026-05-18 06:26:43.327 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-18T06:26:43.467109667Z 2026-05-18 06:26:43.466 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-18T06:26:43.493643733Z 2026-05-18 06:26:43.492 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T06:26:43.699413713Z 2026-05-18 06:26:43.695 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-18T06:26:43.736549649Z 2026-05-18 06:26:43.736 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T06:26:43.736671665Z 2026-05-18 06:26:43.736 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T06:26:44.043062831Z 2026-05-18 06:26:44.041 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-18T06:26:44.074349003Z 2026-05-18 06:26:44.073 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T06:26:44.102420911Z 2026-05-18 06:26:44.101 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-18T06:26:44.127790914Z 2026-05-18 06:26:44.127 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72074547711901696-solr-node1:8983_solr-n_0000000000) starting
2026-05-18T06:26:44.198275961Z 2026-05-18 06:26:44.197 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-18T06:26:44.198433510Z 2026-05-18 06:26:44.198 INFO (OverseerStateUpdate-72074547711901696-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-18T06:26:44.211721123Z 2026-05-18 06:26:44.209 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-18T06:26:44.253866743Z 2026-05-18 06:26:44.253 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-18T06:26:44.800560840Z 2026-05-18 06:26:44.800 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-18T06:26:44.983602966Z 2026-05-18 06:26:44.983 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-18T06:26:45.265003891Z 2026-05-18 06:26:45.264 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T06:26:45.428795058Z 2026-05-18 06:26:45.428 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-18T06:26:45.436096340Z 2026-05-18 06:26:45.435 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_18.request.log
2026-05-18T06:26:45.635059585Z 2026-05-18 06:26:45.488 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-18T06:26:45.638479896Z 2026-05-18 06:26:45.637 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5639ms
2026-05-18T06:26:45.852115585Z 2026-05-18 06:26:45.849 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=45
2026-05-18T06:26:45.928206878Z 2026-05-18 06:26:45.925 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-05-18T06:26:49.552699069Z 2026-05-18 06:26:49.552 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-18T06:26:49.593617767Z 2026-05-18 06:26:49.593 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-18T06:26:54.160149686Z 2026-05-18 06:26:54.159 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-05-18T06:26:54.186396759Z 2026-05-18 06:26:54.185 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-05-18T06:26:54.211380361Z 2026-05-18 06:26:54.210 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-18T06:26:54.524915383Z 2026-05-18 06:26:54.524 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-18T06:26:54.546740384Z 2026-05-18 06:26:54.545 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-18T06:26:54.553984472Z 2026-05-18 06:26:54.553 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:54.591395713Z 2026-05-18 06:26:54.590 INFO (OverseerStateUpdate-72074547711901696-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-18T06:26:54.591418618Z "core":"test-collection_shard3_replica_n1",
2026-05-18T06:26:54.591421821Z "node_name":"solr-node1:8983_solr",
2026-05-18T06:26:54.591424874Z "base_url":"http://solr-node1:8983/solr",
2026-05-18T06:26:54.591427525Z "collection":"test-collection",
2026-05-18T06:26:54.591430589Z "shard":"shard3",
2026-05-18T06:26:54.591433072Z "state":"down",
2026-05-18T06:26:54.591435554Z "type":"NRT",
2026-05-18T06:26:54.591437953Z "operation":"ADDREPLICA",
2026-05-18T06:26:54.591440673Z "waitForFinalState":"false"}
2026-05-18T06:26:54.598303052Z 2026-05-18 06:26:54.597 INFO (OverseerStateUpdate-72074547711901696-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-18T06:26:54.598321694Z "core":"test-collection_shard2_replica_n2",
2026-05-18T06:26:54.598324899Z "node_name":"solr-node3:8985_solr",
2026-05-18T06:26:54.598327638Z "base_url":"http://solr-node3:8985/solr",
2026-05-18T06:26:54.598330247Z "collection":"test-collection",
2026-05-18T06:26:54.598332853Z "shard":"shard2",
2026-05-18T06:26:54.598335284Z "state":"down",
2026-05-18T06:26:54.598337780Z "type":"NRT",
2026-05-18T06:26:54.598340301Z "operation":"ADDREPLICA",
2026-05-18T06:26:54.598342803Z "waitForFinalState":"false"}
2026-05-18T06:26:54.604887286Z 2026-05-18 06:26:54.604 INFO (OverseerStateUpdate-72074547711901696-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-18T06:26:54.604902682Z "core":"test-collection_shard1_replica_n4",
2026-05-18T06:26:54.604906017Z "node_name":"solr-node2:8984_solr",
2026-05-18T06:26:54.604919591Z "base_url":"http://solr-node2:8984/solr",
2026-05-18T06:26:54.604921758Z "collection":"test-collection",
2026-05-18T06:26:54.604923693Z "shard":"shard1",
2026-05-18T06:26:54.604925551Z "state":"down",
2026-05-18T06:26:54.604927480Z "type":"NRT",
2026-05-18T06:26:54.604929411Z "operation":"ADDREPLICA",
2026-05-18T06:26:54.604931382Z "waitForFinalState":"false"}
2026-05-18T06:26:54.713891751Z 2026-05-18 06:26:54.713 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:54.770545679Z 2026-05-18 06:26:54.770 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-18T06:26:54.816628177Z 2026-05-18 06:26:54.816 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-18T06:26:54.918939338Z 2026-05-18 06:26:54.918 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-18T06:26:55.408713347Z 2026-05-18 06:26:55.406 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-18T06:26:55.408802428Z 2026-05-18 06:26:55.406 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-18T06:26:55.442290512Z 2026-05-18 06:26:55.441 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-18T06:26:57.016861007Z 2026-05-18 06:26:57.015 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T06:26:57.100144349Z 2026-05-18 06:26:57.098 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-18T06:26:57.130166746Z 2026-05-18 06:26:57.128 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-18T06:26:57.132045222Z 2026-05-18 06:26:57.129 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-18T06:26:57.134076744Z 2026-05-18 06:26:57.129 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-18T06:26:57.202974559Z 2026-05-18 06:26:57.202 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-18T06:26:57.209962006Z 2026-05-18 06:26:57.209 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-18T06:26:57.216638259Z 2026-05-18 06:26:57.216 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-18T06:26:57.221623921Z 2026-05-18 06:26:57.221 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-18T06:26:57.243979656Z 2026-05-18 06:26:57.243 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-18T06:26:57.251120493Z 2026-05-18 06:26:57.250 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-18T06:26:57.256155261Z 2026-05-18 06:26:57.255 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-18T06:26:57.257285773Z 2026-05-18 06:26:57.257 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-18T06:26:57.286755075Z 2026-05-18 06:26:57.286 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-18T06:26:57.288268128Z 2026-05-18 06:26:57.287 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-18T06:26:57.289636933Z 2026-05-18 06:26:57.289 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-18T06:26:57.307311662Z 2026-05-18 06:26:57.307 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-18T06:26:57.307335800Z 2026-05-18 06:26:57.307 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-18T06:26:57.307399094Z 2026-05-18 06:26:57.307 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-18T06:26:57.309663223Z 2026-05-18 06:26:57.309 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-18T06:26:57.309706290Z 2026-05-18 06:26:57.309 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-18T06:26:57.311143459Z 2026-05-18 06:26:57.310 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72074547711901696-core_node3-n_0000000000
2026-05-18T06:26:57.322379684Z 2026-05-18 06:26:57.322 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-18T06:26:57.430508255Z 2026-05-18 06:26:57.430 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:57.441321277Z 2026-05-18 06:26:57.440 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-18T06:26:57.466518927Z 2026-05-18 06:26:57.465 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2697
2026-05-18T06:26:57.555919548Z 2026-05-18 06:26:57.555 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1732269,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-18T06:26:39.741037448Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-18T06:26:46.291227183Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-18T06:26:46.291252327Z It should be set to 65000 to avoid operational disruption.
2026-05-18T06:26:46.291255547Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-18T06:26:46.291591742Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-18T06:26:46.291602117Z
2026-05-18T06:26:46.301320483Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-18T06:26:46.308916958Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-18T06:26:46.353058436Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-18T06:26:46.385379823Z WARNING: A command line option has enabled the Security Manager
2026-05-18T06:26:46.385404958Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-18T06:26:47.615832568Z 2026-05-18 06:26:47.612 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-18T06:26:47.888139175Z 2026-05-18 06:26:47.887 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-18T06:26:47.915222026Z 2026-05-18 06:26:47.914 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-18T06:26:47.922639826Z 2026-05-18 06:26:47.921 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-18T06:26:47.922659401Z 2026-05-18 06:26:47.921 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-18T06:26:47.922662177Z 2026-05-18 06:26:47.922 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-18T06:26:47.922664785Z 2026-05-18 06:26:47.922 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-18T06:26:47.922114970Z
2026-05-18T06:26:47.925378815Z 2026-05-18 06:26:47.924 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-18T06:26:47.948619723Z 2026-05-18 06:26:47.948 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-18T06:26:47.972977190Z 2026-05-18 06:26:47.972 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T06:26:48.113316539Z 2026-05-18 06:26:48.112 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-18T06:26:48.133389638Z 2026-05-18 06:26:48.132 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T06:26:48.133449647Z 2026-05-18 06:26:48.133 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T06:26:48.133546234Z 2026-05-18 06:26:48.133 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T06:26:48.243823493Z 2026-05-18 06:26:48.243 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-18T06:26:48.244233646Z 2026-05-18 06:26:48.243 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-18T06:26:48.244316227Z 2026-05-18 06:26:48.243 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-18T06:26:48.277718926Z 2026-05-18 06:26:48.277 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-18T06:26:49.142273972Z 2026-05-18 06:26:49.141 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-18T06:26:49.373578837Z 2026-05-18 06:26:49.372 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-18T06:26:49.375013502Z 2026-05-18 06:26:49.373 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-18T06:26:49.385541090Z 2026-05-18 06:26:49.385 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T06:26:49.396809706Z 2026-05-18 06:26:49.396 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-18T06:26:49.404916149Z 2026-05-18 06:26:49.404 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T06:26:49.405014834Z 2026-05-18 06:26:49.404 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T06:26:49.438576793Z 2026-05-18 06:26:49.437 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-18T06:26:49.452914683Z 2026-05-18 06:26:49.452 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-18T06:26:49.482703805Z 2026-05-18 06:26:49.482 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T06:26:49.519375017Z 2026-05-18 06:26:49.517 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-18T06:26:49.541112135Z 2026-05-18 06:26:49.539 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-18T06:26:49.560970579Z 2026-05-18 06:26:49.553 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-18T06:26:49.594011909Z 2026-05-18 06:26:49.593 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-18T06:26:49.679063832Z 2026-05-18 06:26:49.678 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-18T06:26:50.032979723Z 2026-05-18 06:26:50.030 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-18T06:26:50.219510925Z 2026-05-18 06:26:50.219 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-18T06:26:50.509325047Z 2026-05-18 06:26:50.504 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T06:26:50.650010839Z 2026-05-18 06:26:50.649 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-18T06:26:50.657650646Z 2026-05-18 06:26:50.657 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_18.request.log
2026-05-18T06:26:50.663136205Z 2026-05-18 06:26:50.662 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-18T06:26:50.665144484Z 2026-05-18 06:26:50.664 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4357ms
2026-05-18T06:26:50.829052841Z 2026-05-18 06:26:50.828 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=42
2026-05-18T06:26:50.891855310Z 2026-05-18 06:26:50.891 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-05-18T06:26:54.772921442Z 2026-05-18 06:26:54.772 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-18T06:26:54.832719908Z 2026-05-18 06:26:54.832 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-18T06:26:54.888458876Z 2026-05-18 06:26:54.888 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-18T06:26:55.217247400Z 2026-05-18 06:26:55.216 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-18T06:26:55.217271837Z 2026-05-18 06:26:55.216 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-18T06:26:55.236934391Z 2026-05-18 06:26:55.236 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-18T06:26:55.682356470Z 2026-05-18 06:26:55.681 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T06:26:55.702054909Z 2026-05-18 06:26:55.701 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-18T06:26:55.706107238Z 2026-05-18 06:26:55.705 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-18T06:26:55.706911376Z 2026-05-18 06:26:55.706 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-18T06:26:55.706975024Z 2026-05-18 06:26:55.706 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-18T06:26:55.753024244Z 2026-05-18 06:26:55.752 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-18T06:26:55.756173814Z 2026-05-18 06:26:55.755 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-18T06:26:55.758373274Z 2026-05-18 06:26:55.757 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-18T06:26:55.760027699Z 2026-05-18 06:26:55.759 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-18T06:26:55.771393128Z 2026-05-18 06:26:55.770 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-18T06:26:55.779311371Z 2026-05-18 06:26:55.778 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-18T06:26:55.785367321Z 2026-05-18 06:26:55.784 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-18T06:26:55.785457012Z 2026-05-18 06:26:55.785 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-18T06:26:55.794469884Z 2026-05-18 06:26:55.793 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-18T06:26:55.814374603Z 2026-05-18 06:26:55.813 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-18T06:26:55.816162311Z 2026-05-18 06:26:55.815 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-18T06:26:55.841502070Z 2026-05-18 06:26:55.840 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-18T06:26:55.841534690Z 2026-05-18 06:26:55.841 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-18T06:26:55.841537787Z 2026-05-18 06:26:55.841 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-18T06:26:55.843115259Z 2026-05-18 06:26:55.842 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-18T06:26:55.844132838Z 2026-05-18 06:26:55.843 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-18T06:26:55.845506120Z 2026-05-18 06:26:55.845 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72074547711901699-core_node6-n_0000000000
2026-05-18T06:26:55.855484022Z 2026-05-18 06:26:55.855 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-18T06:26:55.997193352Z 2026-05-18 06:26:55.996 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 205] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:56.001565531Z 2026-05-18 06:26:56.001 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-18T06:26:56.014519301Z 2026-05-18 06:26:56.014 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1243
2026-05-18T06:26:56.114865017Z 2026-05-18 06:26:56.114 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 208] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:56.657155983Z 2026-05-18 06:26:56.656 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:56.962207538Z 2026-05-18 06:26:56.961 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 226] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:57.429981660Z 2026-05-18 06:26:57.429 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:57.556233155Z 2026-05-18 06:26:57.555 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1732666,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-18T06:26:46.079487694Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-18T06:26:46.526720670Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-18T06:26:46.526757434Z It should be set to 65000 to avoid operational disruption.
2026-05-18T06:26:46.526760685Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-18T06:26:46.527040266Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-18T06:26:46.527049947Z
2026-05-18T06:26:46.536769875Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-18T06:26:46.544755201Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-18T06:26:46.594240014Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-18T06:26:46.625515997Z WARNING: A command line option has enabled the Security Manager
2026-05-18T06:26:46.625547524Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-18T06:26:47.724425098Z 2026-05-18 06:26:47.721 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-18T06:26:48.001984268Z 2026-05-18 06:26:48.001 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-18T06:26:48.026521664Z 2026-05-18 06:26:48.026 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-18T06:26:48.033398446Z 2026-05-18 06:26:48.032 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-18T06:26:48.033502982Z 2026-05-18 06:26:48.033 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-18T06:26:48.033642319Z 2026-05-18 06:26:48.033 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-18T06:26:48.033675525Z 2026-05-18 06:26:48.033 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-18T06:26:48.033158572Z
2026-05-18T06:26:48.036436664Z 2026-05-18 06:26:48.036 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-18T06:26:48.058368611Z 2026-05-18 06:26:48.057 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-18T06:26:48.082268095Z 2026-05-18 06:26:48.081 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T06:26:48.220605091Z 2026-05-18 06:26:48.220 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-18T06:26:48.238498764Z 2026-05-18 06:26:48.237 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T06:26:48.238572399Z 2026-05-18 06:26:48.238 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T06:26:48.238698848Z 2026-05-18 06:26:48.238 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-18T06:26:48.347235805Z 2026-05-18 06:26:48.346 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-18T06:26:48.347420419Z 2026-05-18 06:26:48.347 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-18T06:26:48.347547346Z 2026-05-18 06:26:48.347 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-18T06:26:48.383169039Z 2026-05-18 06:26:48.382 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-18T06:26:49.316895453Z 2026-05-18 06:26:49.315 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-18T06:26:49.467296109Z 2026-05-18 06:26:49.466 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-18T06:26:49.468707964Z 2026-05-18 06:26:49.466 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-18T06:26:49.477399119Z 2026-05-18 06:26:49.476 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T06:26:49.482580226Z 2026-05-18 06:26:49.482 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-18T06:26:49.487222016Z 2026-05-18 06:26:49.486 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-18T06:26:49.487275352Z 2026-05-18 06:26:49.486 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-18T06:26:49.519080623Z 2026-05-18 06:26:49.518 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-18T06:26:49.534345418Z 2026-05-18 06:26:49.533 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-18T06:26:49.553044515Z 2026-05-18 06:26:49.552 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-18T06:26:49.558834365Z 2026-05-18 06:26:49.558 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-18T06:26:49.573457043Z 2026-05-18 06:26:49.573 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-18T06:26:49.584619047Z 2026-05-18 06:26:49.584 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-18T06:26:49.594234765Z 2026-05-18 06:26:49.593 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-18T06:26:49.635734994Z 2026-05-18 06:26:49.635 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-18T06:26:49.907393760Z 2026-05-18 06:26:49.906 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-18T06:26:50.088270919Z 2026-05-18 06:26:50.087 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-18T06:26:50.358891398Z 2026-05-18 06:26:50.358 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T06:26:50.480063794Z 2026-05-18 06:26:50.479 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-18T06:26:50.485606100Z 2026-05-18 06:26:50.485 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_18.request.log
2026-05-18T06:26:50.496813666Z 2026-05-18 06:26:50.496 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-18T06:26:50.497798800Z 2026-05-18 06:26:50.497 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3954ms
2026-05-18T06:26:52.991636429Z 2026-05-18 06:26:52.978 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=160
2026-05-18T06:26:53.083987025Z 2026-05-18 06:26:53.083 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=23
2026-05-18T06:26:54.491467404Z 2026-05-18 06:26:54.490 INFO (qtp985324122-59-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=21
2026-05-18T06:26:54.772866073Z 2026-05-18 06:26:54.771 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-18T06:26:54.863325487Z 2026-05-18 06:26:54.862 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-18T06:26:54.976896139Z 2026-05-18 06:26:54.976 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-18T06:26:55.376553681Z 2026-05-18 06:26:55.376 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-18T06:26:55.376578209Z 2026-05-18 06:26:55.376 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-18T06:26:55.399103628Z 2026-05-18 06:26:55.398 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-18T06:26:56.204774511Z 2026-05-18 06:26:56.204 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-18T06:26:56.231007723Z 2026-05-18 06:26:56.230 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-18T06:26:56.239461501Z 2026-05-18 06:26:56.239 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-18T06:26:56.302797344Z 2026-05-18 06:26:56.301 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-18T06:26:56.302824936Z 2026-05-18 06:26:56.302 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-18T06:26:56.456088214Z 2026-05-18 06:26:56.455 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-18T06:26:56.473678860Z 2026-05-18 06:26:56.473 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-18T06:26:56.476368636Z 2026-05-18 06:26:56.476 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-18T06:26:56.478096134Z 2026-05-18 06:26:56.477 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-18T06:26:56.490586375Z 2026-05-18 06:26:56.490 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-18T06:26:56.502696331Z 2026-05-18 06:26:56.502 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-18T06:26:56.508314523Z 2026-05-18 06:26:56.507 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-18T06:26:56.508415514Z 2026-05-18 06:26:56.508 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-18T06:26:56.516367511Z 2026-05-18 06:26:56.515 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-18T06:26:56.523889785Z 2026-05-18 06:26:56.523 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-18T06:26:56.525357906Z 2026-05-18 06:26:56.525 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-18T06:26:56.538760871Z 2026-05-18 06:26:56.538 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-18T06:26:56.538918442Z 2026-05-18 06:26:56.538 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-18T06:26:56.538930065Z 2026-05-18 06:26:56.538 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-18T06:26:56.540015700Z 2026-05-18 06:26:56.539 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-18T06:26:56.540511722Z 2026-05-18 06:26:56.540 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-18T06:26:56.541511862Z 2026-05-18 06:26:56.541 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72074547711901700-core_node5-n_0000000000
2026-05-18T06:26:56.552955758Z 2026-05-18 06:26:56.552 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-18T06:26:56.657205737Z 2026-05-18 06:26:56.656 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:56.660970531Z 2026-05-18 06:26:56.660 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-18T06:26:56.669638100Z 2026-05-18 06:26:56.669 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1899
2026-05-18T06:26:56.962253564Z 2026-05-18 06:26:56.961 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 226] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:57.429915835Z 2026-05-18 06:26:57.429 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:57.497627925Z 2026-05-18 06:26:57.497 INFO (qtp985324122-58-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-18T06:26:57.556156688Z 2026-05-18 06:26:57.555 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-18T06:26:57.560209224Z 2026-05-18 06:26:57.559 INFO (qtp985324122-58-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3054
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1732849,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-18T06:26:46.309037033Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 82 Docker SolrCloud rolling upgrade # in 30464 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 4f4fb700ef54: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 3aa9120e46f5: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 40d16f30db40: Already exists
# 9d5a0b6cb36c: Already exists
# 49518c28c0b3: Already exists
# 41162acdcc29: Already exists
# 2ed0fd558ca3: Already exists
# 8b0357ffbe54: Pulling fs layer
# 0df26b0d5a69: Pulling fs layer
# a26fc87814fe: Pulling fs layer
# 67860c4a05fe: Pulling fs layer
# d1a380e5a7b1: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 67860c4a05fe: Waiting
# d1a380e5a7b1: Waiting
# 4f4fb700ef54: Waiting
# 0df26b0d5a69: Verifying Checksum
# 0df26b0d5a69: Download complete
# a26fc87814fe: Download complete
# 67860c4a05fe: Verifying Checksum
# 67860c4a05fe: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# d1a380e5a7b1: Verifying Checksum
# d1a380e5a7b1: Download complete
# 8b0357ffbe54: Verifying Checksum
# 8b0357ffbe54: Download complete
# 8b0357ffbe54: Pull complete
# 0df26b0d5a69: Pull complete
# a26fc87814fe: Pull complete
# 67860c4a05fe: Pull complete
# d1a380e5a7b1: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:16825c9cb8e05e7a6dd9d4209615781d30bc3137a6017a5254cec93560c7c8ba
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 01dd5fbe11936c0ad0879ff9d09132e17f2c83af655311d31c3056637c2b8ffb
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# e1e5c6418c709585da07ce9863ece39e8dec02f55fe116f4fec91659b5d38b57
# Starting solr-node2 connected to first node's ZooKeeper
# 613bca23bda59090f7f38e1f3e9197d6bf43a2e417576912830cd5f46d96f108
# Started solr-node3 connected to first node's ZooKeeper
# c4aff0d28e964a9cf439d026ea69e520f8f31cf2bd8a2c95ed497a1b66854c58
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-18 06:26:54.240; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-18 06:26:54.270; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 83 allowPaths - backup # in 15805 ms
ok 84 snapshot lifecycle # in 4844 ms
ok 85 snapshot list # in 3329 ms
ok 86 snapshot describe # in 3330 ms
ok 87 start solr with ssl # in 16533 ms
ok 88 use different hostname when not checking peer-name # in 24784 ms
ok 89 start solr with ssl and auth # in 14683 ms
ok 90 start solr with client truststore and security manager # in 16620 ms
ok 91 start solr with mTLS needed # in 45727 ms
ok 92 start solr with mTLS wanted # in 34310 ms
ok 93 test keystore reload # in 59102 ms
ok 94 SOLR-11740 check 'solr stop' connection # in 15902 ms
ok 95 stop command for single port # in 18284 ms
ok 96 check stop command doesn't hang # in 218 ms
ok 97 SOLR-16976 solr starts with remote JMX enabled # in 8105 ms
ok 98 deprecated system properties converted to modern properties # in 195021 ms
ok 99 start with custom jetty options # in 8218 ms
ok 100 bootstrapping a configset # in 9305 ms
ok 101 status detects locally running solr # in 9521 ms
ok 102 status with --solr-url from user # in 7935 ms
ok 103 status with --port from user # in 8170 ms
ok 104 multiple connection options are prevented # in 859 ms
ok 105 status with invalid --solr-url from user # in 1246 ms
ok 106 status with --short format # in 195976 ms
ok 107 searching solr via locally executed streaming expression # in 1723 ms
ok 108 searching solr via remotely executed streaming expression # in 1288 ms
ok 109 variable interpolation # in 1705 ms
ok 110 searching solr without credentials fails with error # in 1141 ms
ok 111 --version returns Solr version # in 736 ms
ok 112 version as direct tool call still runs # in 756 ms
ok 113 short help # in 789 ms
ok 114 short help is inferred # in 797 ms
ok 115 long help # in 798 ms
ok 116 running subcommands with zk is prevented # in 731 ms
ok 117 listing out files # in 2145 ms
ok 118 connecting to solr via various solr urls and zk hosts # in 7945 ms
ok 119 copying files around # in 11038 ms
ok 120 upconfig # in 2547 ms
ok 121 SOLR-12429 test upconfig fails with symlink # in 1206 ms
ok 122 downconfig # in 1207 ms
ok 123 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1441 ms
ok 124 env var ZK_HOST is honored # in 3359 ms
ok 125 Cleanup # in 1873 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 15s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/6r34x6y7xxcuy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 211 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/211/
No tests ran.
Build Log:
[...truncated 923 lines...]
BUILD FAILED in 28m 11s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/zmhc3yagdevgu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 210 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/210/
No tests ran.
Build Log:
[...truncated 424 lines...]
# ERROR: Server refused connection at: http://localhost:35851/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:35851/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Posting files to [base] url http://localhost:35851/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:35851/solr/monitors_verbose/update...
#
# ERROR: Server refused connection at: http://localhost:35851/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:35851/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 82 post solr log into solr via cli # in 9182 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 29 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-15T16:22:30.332019868Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-15T16:22:30.332055727Z It should be set to 65000 to avoid operational disruption.
2026-05-15T16:22:30.332059171Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-15T16:22:30.332289820Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-15T16:22:30.332301390Z
2026-05-15T16:22:30.344612183Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-15T16:22:30.355404909Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-15T16:22:30.451832404Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-15T16:22:30.587796270Z WARNING: A command line option has enabled the Security Manager
2026-05-15T16:22:30.587826508Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-15T16:22:32.825865424Z 2026-05-15 16:22:32.820 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-15T16:22:33.432818995Z 2026-05-15 16:22:33.432 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-15T16:22:33.498393783Z 2026-05-15 16:22:33.497 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-15T16:22:33.515433082Z 2026-05-15 16:22:33.514 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-15T16:22:33.515505812Z 2026-05-15 16:22:33.515 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-15T16:22:33.515619397Z 2026-05-15 16:22:33.515 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-15T16:22:33.515734575Z 2026-05-15 16:22:33.515 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-15T16:22:33.515259460Z
2026-05-15T16:22:33.519506623Z 2026-05-15 16:22:33.519 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-15T16:22:33.588966659Z 2026-05-15 16:22:33.588 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-15T16:22:33.644303327Z 2026-05-15 16:22:33.643 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-15T16:22:33.644480725Z 2026-05-15 16:22:33.643 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-15T16:22:33.738250761Z 2026-05-15 16:22:33.737 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-15T16:22:35.370673308Z 2026-05-15 16:22:35.369 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-15T16:22:36.326510569Z 2026-05-15 16:22:36.325 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-15T16:22:36.342769143Z 2026-05-15 16:22:36.342 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-15T16:22:36.342788116Z 2026-05-15 16:22:36.342 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-15T16:22:36.349055630Z 2026-05-15 16:22:36.348 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-15T16:22:36.349070472Z 2026-05-15 16:22:36.348 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-15T16:22:36.455618719Z 2026-05-15 16:22:36.455 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-15T16:22:36.852077130Z 2026-05-15 16:22:36.851 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-15T16:22:36.884846121Z 2026-05-15 16:22:36.884 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T16:22:36.962866269Z 2026-05-15 16:22:36.962 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-15T16:22:36.993262401Z 2026-05-15 16:22:36.992 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T16:22:36.993356233Z 2026-05-15 16:22:36.992 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T16:22:37.130452877Z 2026-05-15 16:22:37.129 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-15T16:22:37.170519256Z 2026-05-15 16:22:37.169 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T16:22:37.183729619Z 2026-05-15 16:22:37.183 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-15T16:22:37.193527714Z 2026-05-15 16:22:37.193 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72059903898681344-solr-node1:8983_solr-n_0000000000) starting
2026-05-15T16:22:37.321246167Z 2026-05-15 16:22:37.320 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-15T16:22:37.321288763Z 2026-05-15 16:22:37.320 INFO (OverseerStateUpdate-72059903898681344-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-15T16:22:37.385536647Z 2026-05-15 16:22:37.385 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-15T16:22:37.521333263Z 2026-05-15 16:22:37.520 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-15T16:22:38.209333359Z 2026-05-15 16:22:38.208 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-15T16:22:38.543424974Z 2026-05-15 16:22:38.542 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-15T16:22:38.946832537Z 2026-05-15 16:22:38.946 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T16:22:39.370410083Z 2026-05-15 16:22:39.369 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-15T16:22:39.506170003Z 2026-05-15 16:22:39.504 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_15.request.log
2026-05-15T16:22:39.510388831Z 2026-05-15 16:22:39.509 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-15T16:22:39.513045396Z 2026-05-15 16:22:39.512 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @9158ms
2026-05-15T16:22:40.275660827Z 2026-05-15 16:22:40.274 INFO (qtp985324122-29-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=105
2026-05-15T16:22:41.010523561Z 2026-05-15 16:22:41.010 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19
2026-05-15T16:22:45.600929194Z 2026-05-15 16:22:45.600 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-05-15T16:22:52.893223793Z 2026-05-15 16:22:52.892 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=44
2026-05-15T16:22:53.037932384Z 2026-05-15 16:22:53.037 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=54
2026-05-15T16:22:53.076630017Z 2026-05-15 16:22:53.076 INFO (qtp985324122-29-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=5
2026-05-15T16:22:53.705372375Z 2026-05-15 16:22:53.704 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=20
2026-05-15T16:22:53.757319061Z 2026-05-15 16:22:53.756 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-15T16:22:53.807528657Z 2026-05-15 16:22:53.807 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-15T16:22:53.832036203Z 2026-05-15 16:22:53.831 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:53.957511863Z 2026-05-15 16:22:53.957 INFO (OverseerStateUpdate-72059903898681344-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-15T16:22:53.957534019Z "core":"test-collection_shard3_replica_n1",
2026-05-15T16:22:53.957536735Z "node_name":"solr-node1:8983_solr",
2026-05-15T16:22:53.957538894Z "base_url":"http://solr-node1:8983/solr",
2026-05-15T16:22:53.957540960Z "collection":"test-collection",
2026-05-15T16:22:53.957543834Z "shard":"shard3",
2026-05-15T16:22:53.957545820Z "state":"down",
2026-05-15T16:22:53.957547763Z "type":"NRT",
2026-05-15T16:22:53.957549751Z "operation":"ADDREPLICA",
2026-05-15T16:22:53.957552031Z "waitForFinalState":"false"}
2026-05-15T16:22:53.985968399Z 2026-05-15 16:22:53.984 INFO (OverseerStateUpdate-72059903898681344-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-15T16:22:53.985985569Z "core":"test-collection_shard2_replica_n2",
2026-05-15T16:22:53.985988213Z "node_name":"solr-node3:8985_solr",
2026-05-15T16:22:53.985990416Z "base_url":"http://solr-node3:8985/solr",
2026-05-15T16:22:53.985992723Z "collection":"test-collection",
2026-05-15T16:22:53.985994733Z "shard":"shard2",
2026-05-15T16:22:53.985996631Z "state":"down",
2026-05-15T16:22:53.985998780Z "type":"NRT",
2026-05-15T16:22:53.986000764Z "operation":"ADDREPLICA",
2026-05-15T16:22:53.986002719Z "waitForFinalState":"false"}
2026-05-15T16:22:53.994703900Z 2026-05-15 16:22:53.993 INFO (OverseerStateUpdate-72059903898681344-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-15T16:22:53.994722569Z "core":"test-collection_shard1_replica_n4",
2026-05-15T16:22:53.994735197Z "node_name":"solr-node2:8984_solr",
2026-05-15T16:22:53.994737463Z "base_url":"http://solr-node2:8984/solr",
2026-05-15T16:22:53.994739486Z "collection":"test-collection",
2026-05-15T16:22:53.994741498Z "shard":"shard1",
2026-05-15T16:22:53.994743947Z "state":"down",
2026-05-15T16:22:53.994745845Z "type":"NRT",
2026-05-15T16:22:53.994747740Z "operation":"ADDREPLICA",
2026-05-15T16:22:53.994749756Z "waitForFinalState":"false"}
2026-05-15T16:22:54.107911835Z 2026-05-15 16:22:54.107 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:54.255534627Z 2026-05-15 16:22:54.255 INFO (qtp985324122-28-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-15T16:22:54.366823096Z 2026-05-15 16:22:54.366 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-15T16:22:54.504918310Z 2026-05-15 16:22:54.504 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-15T16:22:55.658754750Z 2026-05-15 16:22:55.658 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-15T16:22:55.658777902Z 2026-05-15 16:22:55.658 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-15T16:22:55.770700732Z 2026-05-15 16:22:55.769 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-15T16:22:58.143710376Z 2026-05-15 16:22:58.143 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T16:22:58.196652992Z 2026-05-15 16:22:58.195 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-15T16:22:58.203258487Z 2026-05-15 16:22:58.202 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-15T16:22:58.204529589Z 2026-05-15 16:22:58.204 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-15T16:22:58.204608672Z 2026-05-15 16:22:58.204 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-15T16:22:58.307232260Z 2026-05-15 16:22:58.306 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-15T16:22:58.311217027Z 2026-05-15 16:22:58.310 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-15T16:22:58.313999624Z 2026-05-15 16:22:58.313 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-15T16:22:58.322001602Z 2026-05-15 16:22:58.321 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-15T16:22:58.335702328Z 2026-05-15 16:22:58.335 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-15T16:22:58.342792888Z 2026-05-15 16:22:58.342 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-15T16:22:58.351152768Z 2026-05-15 16:22:58.350 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-15T16:22:58.351167419Z 2026-05-15 16:22:58.350 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-15T16:22:58.359303493Z 2026-05-15 16:22:58.359 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-15T16:22:58.371447450Z 2026-05-15 16:22:58.370 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-15T16:22:58.375740912Z 2026-05-15 16:22:58.375 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-15T16:22:58.398782212Z 2026-05-15 16:22:58.398 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-15T16:22:58.398799780Z 2026-05-15 16:22:58.398 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-15T16:22:58.398803388Z 2026-05-15 16:22:58.398 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-15T16:22:58.400671114Z 2026-05-15 16:22:58.400 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-15T16:22:58.401625863Z 2026-05-15 16:22:58.401 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-15T16:22:58.404199682Z 2026-05-15 16:22:58.403 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72059903898681344-core_node3-n_0000000000
2026-05-15T16:22:58.410814074Z 2026-05-15 16:22:58.410 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-15T16:22:58.418915697Z 2026-05-15 16:22:58.418 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-15T16:22:58.431216967Z 2026-05-15 16:22:58.430 INFO (qtp985324122-28-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=4177
2026-05-15T16:22:58.447113266Z 2026-05-15 16:22:58.446 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-15T16:22:58.526561925Z 2026-05-15 16:22:58.526 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:58.529088541Z 2026-05-15 16:22:58.528 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=4803
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1457614,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-15T16:22:29.243323028Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-15T16:22:41.905028544Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-15T16:22:41.905055235Z It should be set to 65000 to avoid operational disruption.
2026-05-15T16:22:41.905058311Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-15T16:22:41.905405233Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-15T16:22:41.905414380Z
2026-05-15T16:22:41.916773279Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-15T16:22:41.927055447Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-15T16:22:41.987521199Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-15T16:22:42.028451217Z WARNING: A command line option has enabled the Security Manager
2026-05-15T16:22:42.028469962Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-15T16:22:43.431569216Z 2026-05-15 16:22:43.428 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-15T16:22:43.710302531Z 2026-05-15 16:22:43.709 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-15T16:22:43.741491860Z 2026-05-15 16:22:43.741 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-15T16:22:43.750735583Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-15T16:22:43.751084812Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-15T16:22:43.751095116Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-15T16:22:43.751097761Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-15T16:22:43.750534382Z
2026-05-15T16:22:43.753704303Z 2026-05-15 16:22:43.753 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-15T16:22:43.796934258Z 2026-05-15 16:22:43.796 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-15T16:22:43.848423068Z 2026-05-15 16:22:43.847 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T16:22:44.010063620Z 2026-05-15 16:22:44.009 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-15T16:22:44.032428278Z 2026-05-15 16:22:44.032 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T16:22:44.032546986Z 2026-05-15 16:22:44.032 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T16:22:44.032677206Z 2026-05-15 16:22:44.032 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T16:22:44.147731736Z 2026-05-15 16:22:44.147 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-15T16:22:44.147994179Z 2026-05-15 16:22:44.147 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-15T16:22:44.148101993Z 2026-05-15 16:22:44.147 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-15T16:22:44.201809785Z 2026-05-15 16:22:44.201 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-15T16:22:45.311567614Z 2026-05-15 16:22:45.310 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-15T16:22:45.467617852Z 2026-05-15 16:22:45.467 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-15T16:22:45.469052756Z 2026-05-15 16:22:45.467 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-15T16:22:45.477902421Z 2026-05-15 16:22:45.477 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T16:22:45.483193337Z 2026-05-15 16:22:45.482 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-15T16:22:45.486055340Z 2026-05-15 16:22:45.485 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T16:22:45.486067556Z 2026-05-15 16:22:45.485 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T16:22:45.527769716Z 2026-05-15 16:22:45.527 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-15T16:22:45.537995338Z 2026-05-15 16:22:45.537 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-15T16:22:45.560924181Z 2026-05-15 16:22:45.560 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T16:22:45.577611649Z 2026-05-15 16:22:45.577 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-15T16:22:45.589533356Z 2026-05-15 16:22:45.588 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-15T16:22:45.602136236Z 2026-05-15 16:22:45.601 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-05-15T16:22:45.664474950Z 2026-05-15 16:22:45.663 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-15T16:22:46.068796963Z 2026-05-15 16:22:46.068 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-15T16:22:46.339950000Z 2026-05-15 16:22:46.339 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-15T16:22:46.635498919Z 2026-05-15 16:22:46.635 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T16:22:46.792433280Z 2026-05-15 16:22:46.791 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-15T16:22:46.798014409Z 2026-05-15 16:22:46.797 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_15.request.log
2026-05-15T16:22:46.801769735Z 2026-05-15 16:22:46.801 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-15T16:22:46.803150758Z 2026-05-15 16:22:46.802 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4877ms
2026-05-15T16:22:47.067886583Z 2026-05-15 16:22:47.066 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=44
2026-05-15T16:22:47.143134589Z 2026-05-15 16:22:47.142 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-05-15T16:22:54.254149485Z 2026-05-15 16:22:54.253 INFO (qtp985324122-28-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-15T16:22:54.325512012Z 2026-05-15 16:22:54.325 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-15T16:22:54.427761192Z 2026-05-15 16:22:54.427 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-15T16:22:55.026501743Z 2026-05-15 16:22:55.025 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-15T16:22:55.026519387Z 2026-05-15 16:22:55.026 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-15T16:22:55.075155350Z 2026-05-15 16:22:55.074 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-15T16:22:55.720372206Z 2026-05-15 16:22:55.719 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T16:22:55.749705878Z 2026-05-15 16:22:55.749 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-15T16:22:55.759399830Z 2026-05-15 16:22:55.758 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-15T16:22:55.760531433Z 2026-05-15 16:22:55.759 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-15T16:22:55.760597758Z 2026-05-15 16:22:55.760 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-15T16:22:55.815611644Z 2026-05-15 16:22:55.815 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-15T16:22:55.819783634Z 2026-05-15 16:22:55.819 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-15T16:22:55.822389201Z 2026-05-15 16:22:55.822 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-15T16:22:55.824797847Z 2026-05-15 16:22:55.824 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-15T16:22:55.841841491Z 2026-05-15 16:22:55.841 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-15T16:22:55.850462069Z 2026-05-15 16:22:55.850 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-15T16:22:55.856821993Z 2026-05-15 16:22:55.856 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-15T16:22:55.856852501Z 2026-05-15 16:22:55.856 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-15T16:22:55.863961753Z 2026-05-15 16:22:55.863 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-15T16:22:55.888285784Z 2026-05-15 16:22:55.887 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-15T16:22:55.889578988Z 2026-05-15 16:22:55.889 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-15T16:22:55.914642969Z 2026-05-15 16:22:55.914 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-15T16:22:55.914799994Z 2026-05-15 16:22:55.914 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-15T16:22:55.914812069Z 2026-05-15 16:22:55.914 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-15T16:22:55.916158538Z 2026-05-15 16:22:55.915 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-15T16:22:55.916518899Z 2026-05-15 16:22:55.916 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-15T16:22:55.917750925Z 2026-05-15 16:22:55.917 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72059903898681348-core_node6-n_0000000000
2026-05-15T16:22:55.937144288Z 2026-05-15 16:22:55.936 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-15T16:22:56.044490181Z 2026-05-15 16:22:56.044 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:56.048152616Z 2026-05-15 16:22:56.047 INFO (qtp985324122-28-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-15T16:22:56.058576648Z 2026-05-15 16:22:56.058 INFO (qtp985324122-28-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1806
2026-05-15T16:22:56.161561369Z 2026-05-15 16:22:56.161 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:58.416031078Z 2026-05-15 16:22:58.415 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:58.526717514Z 2026-05-15 16:22:58.526 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1458025,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-15T16:22:41.194129254Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-15T16:22:41.904992740Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-15T16:22:41.905029126Z It should be set to 65000 to avoid operational disruption.
2026-05-15T16:22:41.905032451Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-15T16:22:41.905280439Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-15T16:22:41.905289945Z
2026-05-15T16:22:41.916173928Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-15T16:22:41.928941200Z [0.004s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-15T16:22:41.987585643Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-15T16:22:42.028382506Z WARNING: A command line option has enabled the Security Manager
2026-05-15T16:22:42.028410428Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-15T16:22:43.431146643Z 2026-05-15 16:22:43.428 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-15T16:22:43.716281283Z 2026-05-15 16:22:43.715 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-15T16:22:43.741702523Z 2026-05-15 16:22:43.741 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-15T16:22:43.751200653Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-15T16:22:43.751214402Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-15T16:22:43.751217022Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-15T16:22:43.751220037Z 2026-05-15 16:22:43.750 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-15T16:22:43.750600842Z
2026-05-15T16:22:43.753837301Z 2026-05-15 16:22:43.753 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-15T16:22:43.796967800Z 2026-05-15 16:22:43.796 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-15T16:22:43.849355227Z 2026-05-15 16:22:43.848 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T16:22:44.010230583Z 2026-05-15 16:22:44.009 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-15T16:22:44.031521582Z 2026-05-15 16:22:44.030 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T16:22:44.031597535Z 2026-05-15 16:22:44.031 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T16:22:44.031774325Z 2026-05-15 16:22:44.031 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T16:22:44.147466027Z 2026-05-15 16:22:44.146 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-15T16:22:44.148006931Z 2026-05-15 16:22:44.147 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-15T16:22:44.148156319Z 2026-05-15 16:22:44.147 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-15T16:22:44.201781177Z 2026-05-15 16:22:44.201 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-15T16:22:45.231740946Z 2026-05-15 16:22:45.231 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-15T16:22:45.458334600Z 2026-05-15 16:22:45.457 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-15T16:22:45.460592331Z 2026-05-15 16:22:45.457 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-15T16:22:45.473610977Z 2026-05-15 16:22:45.473 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T16:22:45.480792438Z 2026-05-15 16:22:45.480 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-15T16:22:45.485303569Z 2026-05-15 16:22:45.484 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T16:22:45.485367701Z 2026-05-15 16:22:45.485 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T16:22:45.527724460Z 2026-05-15 16:22:45.527 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-15T16:22:45.538101513Z 2026-05-15 16:22:45.537 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-15T16:22:45.562225201Z 2026-05-15 16:22:45.561 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T16:22:45.577653693Z 2026-05-15 16:22:45.577 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-15T16:22:45.588305608Z 2026-05-15 16:22:45.587 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-15T16:22:45.601901849Z 2026-05-15 16:22:45.601 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-05-15T16:22:45.664521474Z 2026-05-15 16:22:45.663 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-15T16:22:46.068796015Z 2026-05-15 16:22:46.068 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-15T16:22:46.340104135Z 2026-05-15 16:22:46.339 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-15T16:22:46.635087411Z 2026-05-15 16:22:46.634 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T16:22:46.791615765Z 2026-05-15 16:22:46.790 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-15T16:22:46.797985721Z 2026-05-15 16:22:46.797 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_15.request.log
2026-05-15T16:22:46.801766587Z 2026-05-15 16:22:46.801 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-15T16:22:46.803125023Z 2026-05-15 16:22:46.802 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4878ms
2026-05-15T16:22:49.612040234Z 2026-05-15 16:22:49.603 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=78
2026-05-15T16:22:49.680327943Z 2026-05-15 16:22:49.679 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=20
2026-05-15T16:22:54.269223436Z 2026-05-15 16:22:54.267 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-15T16:22:54.346633296Z 2026-05-15 16:22:54.346 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-15T16:22:54.426420431Z 2026-05-15 16:22:54.425 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-15T16:22:55.035639611Z 2026-05-15 16:22:55.035 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-15T16:22:55.035663652Z 2026-05-15 16:22:55.035 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-15T16:22:55.074968152Z 2026-05-15 16:22:55.074 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-15T16:22:55.722256218Z 2026-05-15 16:22:55.721 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T16:22:55.749593210Z 2026-05-15 16:22:55.749 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-15T16:22:55.759394531Z 2026-05-15 16:22:55.758 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-15T16:22:55.760493244Z 2026-05-15 16:22:55.759 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-15T16:22:55.760550216Z 2026-05-15 16:22:55.760 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-15T16:22:55.814991171Z 2026-05-15 16:22:55.814 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-15T16:22:55.818155168Z 2026-05-15 16:22:55.817 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-15T16:22:55.820079156Z 2026-05-15 16:22:55.819 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-15T16:22:55.822513890Z 2026-05-15 16:22:55.822 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-15T16:22:55.841798598Z 2026-05-15 16:22:55.841 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-15T16:22:55.849625774Z 2026-05-15 16:22:55.849 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-15T16:22:55.856881812Z 2026-05-15 16:22:55.856 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-15T16:22:55.856923791Z 2026-05-15 16:22:55.856 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-15T16:22:55.863215517Z 2026-05-15 16:22:55.862 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-15T16:22:55.888899138Z 2026-05-15 16:22:55.888 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-15T16:22:55.890224232Z 2026-05-15 16:22:55.889 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-15T16:22:55.914776531Z 2026-05-15 16:22:55.914 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-15T16:22:55.914783890Z 2026-05-15 16:22:55.914 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-15T16:22:55.914786688Z 2026-05-15 16:22:55.914 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-15T16:22:55.916627658Z 2026-05-15 16:22:55.916 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-15T16:22:55.917459848Z 2026-05-15 16:22:55.917 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-15T16:22:55.918495297Z 2026-05-15 16:22:55.918 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72059903898681347-core_node5-n_0000000000
2026-05-15T16:22:55.933367104Z 2026-05-15 16:22:55.932 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-15T16:22:56.044210593Z 2026-05-15 16:22:56.043 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:56.048066330Z 2026-05-15 16:22:56.047 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-15T16:22:56.058347277Z 2026-05-15 16:22:56.058 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1806
2026-05-15T16:22:56.162026545Z 2026-05-15 16:22:56.161 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:58.415922758Z 2026-05-15 16:22:58.415 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T16:22:58.526597510Z 2026-05-15 16:22:58.526 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1458155,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-15T16:22:41.414999113Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 33135 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:7b9bb30f4dc4d340f52255f6cab0a7e3bc0bf4e6c8f5b0d5e0744356ef86b458
# Status: Image is up to date for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 241765af35e954f0d0c4b276a2481823836dba20b6a8794cc2cb31d24c6c3c4f
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 7b256613583cc2d5ad747092de8aef692fbbf9929bb95c80141cf9108506f438
# Starting solr-node2 connected to first node's ZooKeeper
# 578e8e5144cb6661b822ec468ce1c04a3734d993d1d08f33a220883567834858
# Started solr-node3 connected to first node's ZooKeeper
# 8d6ce433f0356088d4b73bf98a3f5a28d2cc219a2eabf74e632d0d22271d8851
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-15 16:22:53.291; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-15 16:22:53.386; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 15187 ms
ok 85 snapshot lifecycle # in 4713 ms
ok 86 snapshot list # in 3297 ms
ok 87 snapshot describe # in 3298 ms
ok 88 start solr with ssl # in 15955 ms
ok 89 use different hostname when not checking peer-name # in 24414 ms
ok 90 start solr with ssl and auth # in 14272 ms
ok 91 start solr with client truststore and security manager # in 16089 ms
ok 92 start solr with mTLS needed # in 45604 ms
ok 93 start solr with mTLS wanted # in 34906 ms
ok 94 test keystore reload # in 57897 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15770 ms
ok 96 stop command for single port # in 18112 ms
ok 97 check stop command doesn't hang # in 213 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8244 ms
ok 99 deprecated system properties converted to modern properties # in 194561 ms
ok 100 start with custom jetty options # in 8003 ms
ok 101 bootstrapping a configset # in 9158 ms
ok 102 status detects locally running solr # in 9388 ms
ok 103 status with --solr-url from user # in 7918 ms
ok 104 status with --port from user # in 7983 ms
ok 105 multiple connection options are prevented # in 843 ms
ok 106 status with invalid --solr-url from user # in 1222 ms
ok 107 status with --short format # in 195718 ms
ok 108 searching solr via locally executed streaming expression # in 1708 ms
ok 109 searching solr via remotely executed streaming expression # in 1270 ms
ok 110 variable interpolation # in 1664 ms
ok 111 searching solr without credentials fails with error # in 1112 ms
ok 112 --version returns Solr version # in 724 ms
ok 113 version as direct tool call still runs # in 722 ms
ok 114 short help # in 762 ms
ok 115 short help is inferred # in 754 ms
ok 116 long help # in 762 ms
ok 117 running subcommands with zk is prevented # in 702 ms
ok 118 listing out files # in 2156 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7698 ms
ok 120 copying files around # in 10885 ms
ok 121 upconfig # in 2575 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1182 ms
ok 123 downconfig # in 1183 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1432 ms
ok 125 env var ZK_HOST is honored # in 3160 ms
ok 126 Cleanup # in 1841 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 33s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vi4h6xumkait4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 209 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/209/
No tests ran.
Build Log:
[...truncated 432 lines...]
# ERROR: IOException occurred when talking to server at: http://localhost:43935/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:43935/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:511)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.nio.channels.ClosedChannelException
# at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2141)
# at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1179)
# at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:389)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:526)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:251)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:216)
# at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
# at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:468)
# at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
# at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
# at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.picRunTask(AdaptiveExecutionStrategy.java:473)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:399)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Posting files to [base] url http://localhost:43935/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# <?xml version="1.0" encoding="UTF-8"?>
# <response>
#
# <lst name="responseHeader">
# <int name="rf">1</int>
# <int name="status">0</int>
# <int name="QTime">2353</int>
# </lst>
# </response>
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:43935/solr/monitors_verbose/update...
#
# ERROR: IOException occurred when talking to server at: http://localhost:43935/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:43935/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:511)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.nio.channels.ClosedChannelException
# at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2141)
# at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1179)
# at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:389)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:526)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:251)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:216)
# at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
# at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:468)
# at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
# at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
# at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.picRunTask(AdaptiveExecutionStrategy.java:473)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:399)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 82 post solr log into solr via cli # in 9501 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 12 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 12 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 18 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-15T06:05:55.246939433Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-15T06:05:55.246975419Z It should be set to 65000 to avoid operational disruption.
2026-05-15T06:05:55.246978509Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-15T06:05:55.247228938Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-15T06:05:55.247240643Z
2026-05-15T06:05:55.256547688Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-15T06:05:55.265471520Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-15T06:05:55.306166216Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-15T06:05:55.340715113Z WARNING: A command line option has enabled the Security Manager
2026-05-15T06:05:55.340735531Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-15T06:05:56.445094510Z 2026-05-15 06:05:56.441 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-15T06:05:56.707634968Z 2026-05-15 06:05:56.707 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-15T06:05:56.731290171Z 2026-05-15 06:05:56.730 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-15T06:05:56.738045631Z 2026-05-15 06:05:56.737 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-15T06:05:56.738134868Z 2026-05-15 06:05:56.737 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-15T06:05:56.738216197Z 2026-05-15 06:05:56.737 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-15T06:05:56.738338107Z 2026-05-15 06:05:56.737 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-15T06:05:56.737864247Z
2026-05-15T06:05:56.740745297Z 2026-05-15 06:05:56.740 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-15T06:05:56.761444334Z 2026-05-15 06:05:56.761 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-15T06:05:56.774149369Z 2026-05-15 06:05:56.773 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-15T06:05:56.774210558Z 2026-05-15 06:05:56.773 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-15T06:05:56.808429519Z 2026-05-15 06:05:56.807 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-15T06:05:57.579225702Z 2026-05-15 06:05:57.578 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-15T06:05:57.823648308Z 2026-05-15 06:05:57.823 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-15T06:05:57.828451944Z 2026-05-15 06:05:57.828 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-15T06:05:57.828513149Z 2026-05-15 06:05:57.828 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-15T06:05:57.830940615Z 2026-05-15 06:05:57.830 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-15T06:05:57.830996396Z 2026-05-15 06:05:57.830 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-15T06:05:57.865355572Z 2026-05-15 06:05:57.864 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-15T06:05:58.348253634Z 2026-05-15 06:05:58.345 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-15T06:05:58.363102075Z 2026-05-15 06:05:58.362 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T06:05:58.417453456Z 2026-05-15 06:05:58.416 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-15T06:05:58.447692214Z 2026-05-15 06:05:58.447 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T06:05:58.447797490Z 2026-05-15 06:05:58.447 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T06:05:58.655548504Z 2026-05-15 06:05:58.654 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-15T06:05:58.679223391Z 2026-05-15 06:05:58.678 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T06:05:58.693222114Z 2026-05-15 06:05:58.692 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-15T06:05:58.702076246Z 2026-05-15 06:05:58.701 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72136856819924992-solr-node1:8983_solr-n_0000000000) starting
2026-05-15T06:05:58.782628358Z 2026-05-15 06:05:58.782 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-15T06:05:58.783341801Z 2026-05-15 06:05:58.783 INFO (OverseerStateUpdate-72136856819924992-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-15T06:05:58.800105644Z 2026-05-15 06:05:58.799 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-15T06:05:58.838981058Z 2026-05-15 06:05:58.838 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-15T06:05:59.424602336Z 2026-05-15 06:05:59.421 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-15T06:05:59.612906187Z 2026-05-15 06:05:59.612 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-15T06:05:59.869018157Z 2026-05-15 06:05:59.868 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T06:06:00.448611346Z 2026-05-15 06:06:00.444 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-15T06:06:00.450426599Z 2026-05-15 06:06:00.450 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_15.request.log
2026-05-15T06:06:00.463568156Z 2026-05-15 06:06:00.463 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-15T06:06:00.483650474Z 2026-05-15 06:06:00.483 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5219ms
2026-05-15T06:06:00.696158023Z 2026-05-15 06:06:00.694 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=46
2026-05-15T06:06:00.767198432Z 2026-05-15 06:06:00.766 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-05-15T06:06:04.410874684Z 2026-05-15 06:06:04.410 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-15T06:06:04.705842249Z 2026-05-15 06:06:04.705 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-15T06:06:09.065969207Z 2026-05-15 06:06:09.063 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=22
2026-05-15T06:06:09.105549293Z 2026-05-15 06:06:09.105 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=19
2026-05-15T06:06:09.133088720Z 2026-05-15 06:06:09.132 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-15T06:06:09.738087923Z 2026-05-15 06:06:09.733 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-15T06:06:09.792575846Z 2026-05-15 06:06:09.790 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-15T06:06:09.822176235Z 2026-05-15 06:06:09.820 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:09.926273168Z 2026-05-15 06:06:09.925 INFO (OverseerStateUpdate-72136856819924992-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-15T06:06:09.926296540Z "core":"test-collection_shard3_replica_n1",
2026-05-15T06:06:09.926300056Z "node_name":"solr-node1:8983_solr",
2026-05-15T06:06:09.926302914Z "base_url":"http://solr-node1:8983/solr",
2026-05-15T06:06:09.926305491Z "collection":"test-collection",
2026-05-15T06:06:09.926308037Z "shard":"shard3",
2026-05-15T06:06:09.926310773Z "state":"down",
2026-05-15T06:06:09.926313277Z "type":"NRT",
2026-05-15T06:06:09.926315643Z "operation":"ADDREPLICA",
2026-05-15T06:06:09.926318078Z "waitForFinalState":"false"}
2026-05-15T06:06:09.940532950Z 2026-05-15 06:06:09.940 INFO (OverseerStateUpdate-72136856819924992-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-15T06:06:09.940554380Z "core":"test-collection_shard2_replica_n2",
2026-05-15T06:06:09.940557817Z "node_name":"solr-node3:8985_solr",
2026-05-15T06:06:09.940560794Z "base_url":"http://solr-node3:8985/solr",
2026-05-15T06:06:09.940563609Z "collection":"test-collection",
2026-05-15T06:06:09.940566116Z "shard":"shard2",
2026-05-15T06:06:09.940568559Z "state":"down",
2026-05-15T06:06:09.940571130Z "type":"NRT",
2026-05-15T06:06:09.940573463Z "operation":"ADDREPLICA",
2026-05-15T06:06:09.940575886Z "waitForFinalState":"false"}
2026-05-15T06:06:09.948420409Z 2026-05-15 06:06:09.948 INFO (OverseerStateUpdate-72136856819924992-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-15T06:06:09.948434120Z "core":"test-collection_shard1_replica_n4",
2026-05-15T06:06:09.948436828Z "node_name":"solr-node2:8984_solr",
2026-05-15T06:06:09.948451031Z "base_url":"http://solr-node2:8984/solr",
2026-05-15T06:06:09.948453051Z "collection":"test-collection",
2026-05-15T06:06:09.948454959Z "shard":"shard1",
2026-05-15T06:06:09.948457318Z "state":"down",
2026-05-15T06:06:09.948459126Z "type":"NRT",
2026-05-15T06:06:09.948461285Z "operation":"ADDREPLICA",
2026-05-15T06:06:09.948463197Z "waitForFinalState":"false"}
2026-05-15T06:06:10.061203075Z 2026-05-15 06:06:10.060 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:10.318525647Z 2026-05-15 06:06:10.318 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-15T06:06:10.444061177Z 2026-05-15 06:06:10.438 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-15T06:06:10.569593016Z 2026-05-15 06:06:10.568 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-15T06:06:11.594046396Z 2026-05-15 06:06:11.592 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-15T06:06:11.594086546Z 2026-05-15 06:06:11.592 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-15T06:06:11.615949015Z 2026-05-15 06:06:11.615 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-15T06:06:12.553401931Z 2026-05-15 06:06:12.552 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T06:06:12.697643733Z 2026-05-15 06:06:12.697 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-15T06:06:12.726742291Z 2026-05-15 06:06:12.726 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-15T06:06:12.730016655Z 2026-05-15 06:06:12.729 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-15T06:06:12.730207552Z 2026-05-15 06:06:12.729 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-15T06:06:12.978256123Z 2026-05-15 06:06:12.977 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-15T06:06:13.005391452Z 2026-05-15 06:06:13.004 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-15T06:06:13.013669472Z 2026-05-15 06:06:13.013 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-15T06:06:13.033040998Z 2026-05-15 06:06:13.032 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-15T06:06:13.115017731Z 2026-05-15 06:06:13.114 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-15T06:06:13.151419618Z 2026-05-15 06:06:13.150 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-15T06:06:13.185324028Z 2026-05-15 06:06:13.184 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-15T06:06:13.185346304Z 2026-05-15 06:06:13.184 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-15T06:06:13.245516073Z 2026-05-15 06:06:13.245 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-15T06:06:13.246960088Z 2026-05-15 06:06:13.246 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-15T06:06:13.253608802Z 2026-05-15 06:06:13.253 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-15T06:06:13.349158422Z 2026-05-15 06:06:13.348 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-15T06:06:13.349181571Z 2026-05-15 06:06:13.348 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-15T06:06:13.349216367Z 2026-05-15 06:06:13.348 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-15T06:06:13.358036295Z 2026-05-15 06:06:13.357 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-15T06:06:13.363738989Z 2026-05-15 06:06:13.363 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-15T06:06:13.371952463Z 2026-05-15 06:06:13.371 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72136856819924992-core_node3-n_0000000000
2026-05-15T06:06:13.449956613Z 2026-05-15 06:06:13.449 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-15T06:06:13.575077594Z 2026-05-15 06:06:13.574 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:13.584532197Z 2026-05-15 06:06:13.584 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-15T06:06:13.643249711Z 2026-05-15 06:06:13.642 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3327
2026-05-15T06:06:13.704410256Z 2026-05-15 06:06:13.703 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 248] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3005686,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-15T06:05:55.004130299Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-15T06:06:01.189797630Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-15T06:06:01.189841789Z It should be set to 65000 to avoid operational disruption.
2026-05-15T06:06:01.189845297Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-15T06:06:01.190412431Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-15T06:06:01.190428752Z
2026-05-15T06:06:01.201610848Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-15T06:06:01.210550912Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-15T06:06:01.264311592Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-15T06:06:01.315395406Z WARNING: A command line option has enabled the Security Manager
2026-05-15T06:06:01.315422634Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-15T06:06:02.567037428Z 2026-05-15 06:06:02.563 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-15T06:06:02.842074415Z 2026-05-15 06:06:02.841 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-15T06:06:02.867542453Z 2026-05-15 06:06:02.867 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-15T06:06:02.874712194Z 2026-05-15 06:06:02.874 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-15T06:06:02.874820707Z 2026-05-15 06:06:02.874 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-15T06:06:02.874968340Z 2026-05-15 06:06:02.874 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-15T06:06:02.875101707Z 2026-05-15 06:06:02.874 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-15T06:06:02.874510348Z
2026-05-15T06:06:02.877600931Z 2026-05-15 06:06:02.877 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-15T06:06:02.898838101Z 2026-05-15 06:06:02.898 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-15T06:06:02.922559439Z 2026-05-15 06:06:02.922 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T06:06:03.071758366Z 2026-05-15 06:06:03.071 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-15T06:06:03.093330436Z 2026-05-15 06:06:03.092 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T06:06:03.093553367Z 2026-05-15 06:06:03.092 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T06:06:03.093800394Z 2026-05-15 06:06:03.092 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T06:06:03.204050220Z 2026-05-15 06:06:03.203 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-15T06:06:03.204211041Z 2026-05-15 06:06:03.203 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-15T06:06:03.204501932Z 2026-05-15 06:06:03.204 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-15T06:06:03.238973763Z 2026-05-15 06:06:03.238 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-15T06:06:03.985507286Z 2026-05-15 06:06:03.984 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-15T06:06:04.194696020Z 2026-05-15 06:06:04.183 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-15T06:06:04.197218783Z 2026-05-15 06:06:04.185 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-15T06:06:04.204969393Z 2026-05-15 06:06:04.204 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T06:06:04.220670729Z 2026-05-15 06:06:04.220 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-15T06:06:04.232788505Z 2026-05-15 06:06:04.232 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T06:06:04.233000290Z 2026-05-15 06:06:04.232 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T06:06:04.279710764Z 2026-05-15 06:06:04.278 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-15T06:06:04.307962793Z 2026-05-15 06:06:04.307 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-15T06:06:04.358712512Z 2026-05-15 06:06:04.358 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T06:06:04.376877007Z 2026-05-15 06:06:04.374 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-15T06:06:04.393345299Z 2026-05-15 06:06:04.392 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-15T06:06:04.412026508Z 2026-05-15 06:06:04.411 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-15T06:06:04.534046534Z 2026-05-15 06:06:04.532 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-15T06:06:04.741012517Z 2026-05-15 06:06:04.739 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-15T06:06:05.009076295Z 2026-05-15 06:06:05.007 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-15T06:06:05.192250519Z 2026-05-15 06:06:05.191 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-15T06:06:05.471898110Z 2026-05-15 06:06:05.471 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T06:06:05.638569374Z 2026-05-15 06:06:05.637 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-15T06:06:05.645946075Z 2026-05-15 06:06:05.645 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_15.request.log
2026-05-15T06:06:05.651824862Z 2026-05-15 06:06:05.651 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-15T06:06:05.654459226Z 2026-05-15 06:06:05.654 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4445ms
2026-05-15T06:06:05.814280816Z 2026-05-15 06:06:05.813 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-05-15T06:06:05.870054653Z 2026-05-15 06:06:05.869 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-15T06:06:10.312036329Z 2026-05-15 06:06:10.311 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-15T06:06:10.363665573Z 2026-05-15 06:06:10.363 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-15T06:06:10.426242732Z 2026-05-15 06:06:10.425 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-15T06:06:10.757592973Z 2026-05-15 06:06:10.757 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-15T06:06:10.757618076Z 2026-05-15 06:06:10.757 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-15T06:06:10.777183490Z 2026-05-15 06:06:10.776 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-15T06:06:11.233082127Z 2026-05-15 06:06:11.232 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T06:06:11.252406539Z 2026-05-15 06:06:11.251 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-15T06:06:11.255880644Z 2026-05-15 06:06:11.255 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-15T06:06:11.257118775Z 2026-05-15 06:06:11.256 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-15T06:06:11.257186270Z 2026-05-15 06:06:11.256 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-15T06:06:11.297440696Z 2026-05-15 06:06:11.297 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-15T06:06:11.300386473Z 2026-05-15 06:06:11.300 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-15T06:06:11.302158358Z 2026-05-15 06:06:11.301 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-15T06:06:11.303649936Z 2026-05-15 06:06:11.303 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-15T06:06:11.315033880Z 2026-05-15 06:06:11.314 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-15T06:06:11.322422884Z 2026-05-15 06:06:11.322 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-15T06:06:11.327708838Z 2026-05-15 06:06:11.327 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-15T06:06:11.327793739Z 2026-05-15 06:06:11.327 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-15T06:06:11.335841969Z 2026-05-15 06:06:11.335 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-15T06:06:11.349864358Z 2026-05-15 06:06:11.349 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-15T06:06:11.351355318Z 2026-05-15 06:06:11.351 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-15T06:06:11.380031891Z 2026-05-15 06:06:11.379 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-15T06:06:11.380050368Z 2026-05-15 06:06:11.379 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-15T06:06:11.380337496Z 2026-05-15 06:06:11.379 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-15T06:06:11.381790987Z 2026-05-15 06:06:11.381 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-15T06:06:11.382627902Z 2026-05-15 06:06:11.382 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-15T06:06:11.383940571Z 2026-05-15 06:06:11.383 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72136856819924995-core_node6-n_0000000000
2026-05-15T06:06:11.393443934Z 2026-05-15 06:06:11.393 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-15T06:06:11.504154178Z 2026-05-15 06:06:11.503 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:11.508177485Z 2026-05-15 06:06:11.507 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-15T06:06:11.518238446Z 2026-05-15 06:06:11.517 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1207
2026-05-15T06:06:11.619017574Z 2026-05-15 06:06:11.618 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 209] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:12.460346598Z 2026-05-15 06:06:12.459 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:12.624340657Z 2026-05-15 06:06:12.623 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 227] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:13.575298873Z 2026-05-15 06:06:13.574 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:13.704663193Z 2026-05-15 06:06:13.704 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 248] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3006086,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-15T06:06:00.961419011Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-15T06:06:01.576621262Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-15T06:06:01.576677348Z It should be set to 65000 to avoid operational disruption.
2026-05-15T06:06:01.576682215Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-15T06:06:01.576945149Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-15T06:06:01.576971388Z
2026-05-15T06:06:01.589406308Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-15T06:06:01.600325905Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-15T06:06:01.654931724Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-15T06:06:01.698643245Z WARNING: A command line option has enabled the Security Manager
2026-05-15T06:06:01.698682568Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-15T06:06:02.978688104Z 2026-05-15 06:06:02.975 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-15T06:06:03.253814114Z 2026-05-15 06:06:03.253 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-15T06:06:03.278501507Z 2026-05-15 06:06:03.277 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-15T06:06:03.285514987Z 2026-05-15 06:06:03.284 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-15T06:06:03.286017699Z 2026-05-15 06:06:03.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-15T06:06:03.286039353Z 2026-05-15 06:06:03.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-15T06:06:03.286043278Z 2026-05-15 06:06:03.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-15T06:06:03.285205177Z
2026-05-15T06:06:03.288536000Z 2026-05-15 06:06:03.288 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-15T06:06:03.310764829Z 2026-05-15 06:06:03.310 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-15T06:06:03.336089071Z 2026-05-15 06:06:03.335 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T06:06:03.474132628Z 2026-05-15 06:06:03.473 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-15T06:06:03.491713886Z 2026-05-15 06:06:03.491 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T06:06:03.491916174Z 2026-05-15 06:06:03.491 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T06:06:03.492082054Z 2026-05-15 06:06:03.491 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-15T06:06:03.601251368Z 2026-05-15 06:06:03.600 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-15T06:06:03.601391651Z 2026-05-15 06:06:03.601 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-15T06:06:03.601447206Z 2026-05-15 06:06:03.601 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-15T06:06:03.635087125Z 2026-05-15 06:06:03.634 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-15T06:06:04.452142528Z 2026-05-15 06:06:04.451 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-15T06:06:04.590544333Z 2026-05-15 06:06:04.590 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-15T06:06:04.592276282Z 2026-05-15 06:06:04.590 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-15T06:06:04.600705530Z 2026-05-15 06:06:04.600 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T06:06:04.605613869Z 2026-05-15 06:06:04.605 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-15T06:06:04.610438712Z 2026-05-15 06:06:04.610 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-15T06:06:04.610512805Z 2026-05-15 06:06:04.610 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-15T06:06:04.643455982Z 2026-05-15 06:06:04.642 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-15T06:06:04.651856604Z 2026-05-15 06:06:04.651 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-15T06:06:04.671525605Z 2026-05-15 06:06:04.671 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-15T06:06:04.684521351Z 2026-05-15 06:06:04.684 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-15T06:06:04.696622597Z 2026-05-15 06:06:04.696 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-15T06:06:04.706018381Z 2026-05-15 06:06:04.705 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-15T06:06:04.756583775Z 2026-05-15 06:06:04.756 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-15T06:06:05.019127320Z 2026-05-15 06:06:05.018 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-15T06:06:05.187921459Z 2026-05-15 06:06:05.187 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-15T06:06:05.447524090Z 2026-05-15 06:06:05.447 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T06:06:05.573663605Z 2026-05-15 06:06:05.572 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-15T06:06:05.579349587Z 2026-05-15 06:06:05.579 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_15.request.log
2026-05-15T06:06:05.582810118Z 2026-05-15 06:06:05.582 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-15T06:06:05.583634547Z 2026-05-15 06:06:05.583 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3985ms
2026-05-15T06:06:07.837068451Z 2026-05-15 06:06:07.795 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=195
2026-05-15T06:06:07.937262279Z 2026-05-15 06:06:07.936 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=44
2026-05-15T06:06:09.653521246Z 2026-05-15 06:06:09.653 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=14
2026-05-15T06:06:10.317789455Z 2026-05-15 06:06:10.313 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-15T06:06:10.457926468Z 2026-05-15 06:06:10.457 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-15T06:06:10.661118592Z 2026-05-15 06:06:10.660 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-15T06:06:11.528452194Z 2026-05-15 06:06:11.527 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-15T06:06:11.528482797Z 2026-05-15 06:06:11.528 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-15T06:06:11.552065409Z 2026-05-15 06:06:11.551 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-15T06:06:12.166532162Z 2026-05-15 06:06:12.166 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-15T06:06:12.189622806Z 2026-05-15 06:06:12.189 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-15T06:06:12.207487658Z 2026-05-15 06:06:12.206 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-15T06:06:12.208412764Z 2026-05-15 06:06:12.207 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-15T06:06:12.208425121Z 2026-05-15 06:06:12.208 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-15T06:06:12.256377043Z 2026-05-15 06:06:12.256 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-15T06:06:12.258866387Z 2026-05-15 06:06:12.258 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-15T06:06:12.260442911Z 2026-05-15 06:06:12.260 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-15T06:06:12.261941630Z 2026-05-15 06:06:12.261 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-15T06:06:12.272396005Z 2026-05-15 06:06:12.272 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-15T06:06:12.279232557Z 2026-05-15 06:06:12.278 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-15T06:06:12.284536454Z 2026-05-15 06:06:12.284 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-15T06:06:12.284564818Z 2026-05-15 06:06:12.284 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-15T06:06:12.292833169Z 2026-05-15 06:06:12.292 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-15T06:06:12.311866446Z 2026-05-15 06:06:12.311 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-15T06:06:12.313336287Z 2026-05-15 06:06:12.312 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-15T06:06:12.338268410Z 2026-05-15 06:06:12.336 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-15T06:06:12.338286979Z 2026-05-15 06:06:12.338 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-15T06:06:12.338341684Z 2026-05-15 06:06:12.338 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-15T06:06:12.339645738Z 2026-05-15 06:06:12.339 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-15T06:06:12.340355020Z 2026-05-15 06:06:12.340 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-15T06:06:12.341459229Z 2026-05-15 06:06:12.341 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72136856819924996-core_node5-n_0000000000
2026-05-15T06:06:12.354570147Z 2026-05-15 06:06:12.354 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-15T06:06:12.459895779Z 2026-05-15 06:06:12.459 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:12.512639052Z 2026-05-15 06:06:12.512 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-15T06:06:12.524110036Z 2026-05-15 06:06:12.523 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2213
2026-05-15T06:06:12.624684834Z 2026-05-15 06:06:12.624 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 227] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:13.575342223Z 2026-05-15 06:06:13.574 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:13.679876537Z 2026-05-15 06:06:13.679 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-15T06:06:13.704731519Z 2026-05-15 06:06:13.704 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 248] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-15T06:06:13.706546780Z 2026-05-15 06:06:13.706 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=4041
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3006279,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-15T06:06:01.22501778Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 37575 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# 9cdfe5320de8: Waiting
# 4f4fb700ef54: Waiting
# 3aa9120e46f5: Waiting
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# 87f3bb8db2f4: Already exists
# 7c0b62ebdf15: Already exists
# 5b95479e5f9b: Already exists
# c913227527c9: Already exists
# 256184225294: Pulling fs layer
# d9d93b9bef8f: Pulling fs layer
# f05c096a6ea4: Pulling fs layer
# d6e2fa49c345: Pulling fs layer
# 163ca4889105: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# d6e2fa49c345: Waiting
# 163ca4889105: Waiting
# 4f4fb700ef54: Waiting
# d9d93b9bef8f: Download complete
# f05c096a6ea4: Verifying Checksum
# f05c096a6ea4: Download complete
# d6e2fa49c345: Verifying Checksum
# d6e2fa49c345: Download complete
# 163ca4889105: Verifying Checksum
# 163ca4889105: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 256184225294: Verifying Checksum
# 256184225294: Download complete
# 256184225294: Pull complete
# d9d93b9bef8f: Pull complete
# f05c096a6ea4: Pull complete
# d6e2fa49c345: Pull complete
# 163ca4889105: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:7b9bb30f4dc4d340f52255f6cab0a7e3bc0bf4e6c8f5b0d5e0744356ef86b458
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# c2566c6a6cb0362202e487ccd681e595da4fddf2020f907f6f5d3242e9884c70
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 1a1defd1a6f8bea174fc9b05e0886ebfdd3d721dddf3494227d3627d175b9ccd
# Starting solr-node2 connected to first node's ZooKeeper
# 85c5460036559b6da087f440ba62912a1d6433a76568c130a468f8f6db7d7f38
# Started solr-node3 connected to first node's ZooKeeper
# c0a7c2f1ad62410209aca25372f8691d75f9c8f1421bd42c6b2e2dc0d49b2759
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-15 06:06:09.163; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-15 06:06:09.199; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 15887 ms
ok 85 snapshot lifecycle # in 4931 ms
ok 86 snapshot list # in 3322 ms
ok 87 snapshot describe # in 3394 ms
ok 88 start solr with ssl # in 16436 ms
ok 89 use different hostname when not checking peer-name # in 24548 ms
ok 90 start solr with ssl and auth # in 14750 ms
ok 91 start solr with client truststore and security manager # in 17106 ms
ok 92 start solr with mTLS needed # in 45982 ms
ok 93 start solr with mTLS wanted # in 34513 ms
ok 94 test keystore reload # in 59028 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15877 ms
ok 96 stop command for single port # in 18455 ms
ok 97 check stop command doesn't hang # in 215 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8419 ms
ok 99 deprecated system properties converted to modern properties # in 195028 ms
ok 100 start with custom jetty options # in 8222 ms
ok 101 bootstrapping a configset # in 9261 ms
ok 102 status detects locally running solr # in 9625 ms
ok 103 status with --solr-url from user # in 8132 ms
ok 104 status with --port from user # in 8463 ms
ok 105 multiple connection options are prevented # in 865 ms
ok 106 status with invalid --solr-url from user # in 1212 ms
ok 107 status with --short format # in 195796 ms
ok 108 searching solr via locally executed streaming expression # in 1768 ms
ok 109 searching solr via remotely executed streaming expression # in 1289 ms
ok 110 variable interpolation # in 1741 ms
ok 111 searching solr without credentials fails with error # in 1137 ms
ok 112 --version returns Solr version # in 758 ms
ok 113 version as direct tool call still runs # in 752 ms
ok 114 short help # in 816 ms
ok 115 short help is inferred # in 782 ms
ok 116 long help # in 778 ms
ok 117 running subcommands with zk is prevented # in 710 ms
ok 118 listing out files # in 2133 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7822 ms
ok 120 copying files around # in 11042 ms
ok 121 upconfig # in 2609 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1189 ms
ok 123 downconfig # in 1211 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1429 ms
ok 125 env var ZK_HOST is honored # in 3231 ms
ok 126 Cleanup # in 1852 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 30m 17s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/rhhzifcrtcbfy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 208 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/208/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-05-14 03:48:07.494; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-14 03:48:07.497; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-14 03:48:07.941; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-14 03:48:07.941; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1289 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 42883 is already being used by another process (pid: 1113285)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-14 03:48:09.091; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-14 03:48:09.094; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 17551 ms
ok 11 package with basic auth # in 11932 ms
ok 12 admin UI static content is excepted from basic auth # in 10547 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 158 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 79 ms
ok 16 compressed response can be decompressed and parsed # in 78 ms
ok 17 setting property # in 10991 ms
ok 18 short form of setting property # in 6349 ms
ok 19 connecting to solr via various solr urls and zk hosts # in 9337 ms
ok 20 create for non cloud mode # in 7817 ms
ok 21 create for cloud mode # in 9761 ms
ok 22 multiple connection options are prevented # in 860 ms
ok 23 create collection # in 7644 ms
ok 24 create collection using solrUrl # in 5221 ms
ok 25 create collection using legacy solrUrl # in 5213 ms
ok 26 create collection using Zookeeper # in 5247 ms
ok 27 reject d option with invalid config dir # in 2656 ms
ok 28 accept d option with builtin config # in 5158 ms
ok 29 accept d option with explicit path to config # in 5257 ms
ok 30 accept n option as config name # in 5290 ms
ok 31 allow config reuse when n option specifies same config # in 8988 ms
ok 32 create multisharded collections when s provided # in 5185 ms
ok 33 create replicated collections when rf provided # in 6122 ms
ok 34 can delete collections # in 10613 ms
ok 35 can delete collections with solr-url # in 8294 ms
ok 36 collection delete also deletes zk config # in 7284 ms
ok 37 deletes accompanying zk config with nondefault name # in 7236 ms
ok 38 delete-config option can opt to leave config in zk # in 7186 ms
ok 39 start -e cloud works with --jvm-opts # in 21881 ms
ok 40 start -e cloud works with --no-prompt # in 46125 ms
ok 41 Check export command # in 19080 ms
ok 42 export fails on non cloud mode # in 9415 ms
Waiting for Tika Server to be ready on port 42888
WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed.
WARNING: Docker not available; bypassing test.
ok 43 using curl to extract a single pdf file # in 16602 ms
WARNING: Docker not available; bypassing test.
ok 44 using the bin/solr post tool to extract content from pdf # in 16550 ms
WARNING: Docker not available; bypassing test.
ok 45 using the bin/solr post tool to crawl web site # in 16573 ms
Stopping Tika Server container
ok 46 healthcheck on cloud solr # in 10174 ms
ok 47 healthcheck errors on standalone solr # in 11209 ms
ok 48 solr help flag prints help # in 728 ms
ok 49 solr with no flags prints help # in 731 ms
ok 50 start help flag prints help # in 142 ms
ok 51 start h flag prints help # in 146 ms
ok 52 stop help flag prints help # in 139 ms
ok 53 restart help flag prints help # in 147 ms
ok 54 status help flag prints help # in 803 ms
ok 55 healthcheck help flag prints help # in 764 ms
ok 56 create help flag prints help # in 772 ms
ok 57 delete help flag prints help # in 742 ms
ok 58 zk help flag prints help # in 750 ms
ok 59 auth help flag prints help # in 767 ms
ok 60 assert help flag prints help # in 760 ms
ok 61 post help flag prints help # in 766 ms
ok 62 SQL Module # in 15713 ms
ok 63 icu collation in analysis-extras module # in 12276 ms
ok 64 Check lifecycle of sentiment classification # in 23526 ms
ok 65 lifecycle of package # in 8531 ms
ok 66 deploying and undeploying a collection level package # in 13678 ms
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 68 Affinity placement plugin using sysprop # in 15563 ms
ok 69 Random placement plugin using ENV # in 15311 ms
ok 70 Check help command # in 2327 ms
ok 71 basic post with a type specified # in 5394 ms
ok 72 basic post with collection instead of url parameter # in 4127 ms
ok 73 basic post with solr-url and collection # in 3902 ms
ok 74 basic post WITHOUT a type specified # in 11385 ms
ok 75 crawling a directory as a dry-run # in 790 ms
ok 76 crawling a directory # in 4416 ms
ok 77 crawling a web site # in 34882 ms
ok 78 skipcommit and optimize and delete # in 4926 ms
ok 79 args mode # in 9070 ms
ok 80 stdin mode # in 3831 ms
not ok 81 verbose echo the Solr response # in 11893 ms
# (from function `refute_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 360,
# in test file test/test_post.bats, line 216)
# `refute_output --partial 'ERROR'' failed
#
# -- output should not contain substring --
# substring (1 lines):
# ERROR
# output (47 lines):
# Posting files to [base] url http://localhost:42883/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# <?xml version="1.0" encoding="UTF-8"?>
# <response>
#
# <lst name="responseHeader">
# <int name="rf">1</int>
# <int name="status">0</int>
# <int name="QTime">3108</int>
# </lst>
# </response>
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:42883/solr/monitors_verbose/update...
#
# ERROR: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:511)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.nio.channels.ClosedChannelException
# at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2141)
# at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1179)
# at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:389)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:526)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:251)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:216)
# at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
# at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:468)
# at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
# at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
# at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.picRunTask(AdaptiveExecutionStrategy.java:473)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:399)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Posting files to [base] url http://localhost:42883/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# <?xml version="1.0" encoding="UTF-8"?>
# <response>
#
# <lst name="responseHeader">
# <int name="rf">1</int>
# <int name="status">0</int>
# <int name="QTime">3108</int>
# </lst>
# </response>
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:42883/solr/monitors_verbose/update...
#
# ERROR: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:511)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.nio.channels.ClosedChannelException
# at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2141)
# at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1179)
# at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:389)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:526)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:251)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:216)
# at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
# at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:468)
# at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
# at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
# at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.picRunTask(AdaptiveExecutionStrategy.java:473)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:399)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 82 post solr log into solr via cli # in 9482 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-14T03:58:52.489921273Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-14T03:58:52.489962680Z It should be set to 65000 to avoid operational disruption.
2026-05-14T03:58:52.489967439Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-14T03:58:52.490221965Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-14T03:58:52.490233893Z
2026-05-14T03:58:52.499451669Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-14T03:58:52.506915283Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-14T03:58:52.549277905Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-14T03:58:52.580171633Z WARNING: A command line option has enabled the Security Manager
2026-05-14T03:58:52.580193485Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-14T03:58:53.758239767Z 2026-05-14 03:58:53.754 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-14T03:58:54.030706437Z 2026-05-14 03:58:54.030 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-14T03:58:54.055421273Z 2026-05-14 03:58:54.054 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-14T03:58:54.062164720Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-14T03:58:54.062240329Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-14T03:58:54.062378234Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-14T03:58:54.062528962Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-14T03:58:54.061953044Z
2026-05-14T03:58:54.065007446Z 2026-05-14 03:58:54.064 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-14T03:58:54.086374389Z 2026-05-14 03:58:54.085 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-14T03:58:54.099573459Z 2026-05-14 03:58:54.099 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-14T03:58:54.099635941Z 2026-05-14 03:58:54.099 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-14T03:58:54.133374425Z 2026-05-14 03:58:54.132 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-14T03:58:54.913841196Z 2026-05-14 03:58:54.913 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-14T03:58:55.154995007Z 2026-05-14 03:58:55.154 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-14T03:58:55.159953335Z 2026-05-14 03:58:55.159 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-14T03:58:55.160154173Z 2026-05-14 03:58:55.159 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-14T03:58:55.162376609Z 2026-05-14 03:58:55.162 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-14T03:58:55.162446736Z 2026-05-14 03:58:55.162 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-14T03:58:55.198139991Z 2026-05-14 03:58:55.197 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-14T03:58:55.666316018Z 2026-05-14 03:58:55.664 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-14T03:58:55.679562911Z 2026-05-14 03:58:55.679 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-14T03:58:55.739566107Z 2026-05-14 03:58:55.739 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-14T03:58:55.766486901Z 2026-05-14 03:58:55.765 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-14T03:58:55.766563759Z 2026-05-14 03:58:55.766 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-14T03:58:55.907897480Z 2026-05-14 03:58:55.906 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-14T03:58:55.932636689Z 2026-05-14 03:58:55.932 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-14T03:58:55.945466397Z 2026-05-14 03:58:55.944 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-14T03:58:55.957346057Z 2026-05-14 03:58:55.956 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72130694953107456-solr-node1:8983_solr-n_0000000000) starting
2026-05-14T03:58:56.027515852Z 2026-05-14 03:58:56.026 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-14T03:58:56.028311027Z 2026-05-14 03:58:56.027 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-14T03:58:56.038568226Z 2026-05-14 03:58:56.038 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-14T03:58:56.082100115Z 2026-05-14 03:58:56.081 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-14T03:58:56.441979105Z 2026-05-14 03:58:56.441 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-14T03:58:56.607074399Z 2026-05-14 03:58:56.606 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-14T03:58:56.868587264Z 2026-05-14 03:58:56.868 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-14T03:58:57.001656575Z 2026-05-14 03:58:57.000 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-14T03:58:57.007259820Z 2026-05-14 03:58:57.006 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_14.request.log
2026-05-14T03:58:57.011375645Z 2026-05-14 03:58:57.010 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-14T03:58:57.013161905Z 2026-05-14 03:58:57.012 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4507ms
2026-05-14T03:58:57.297774553Z 2026-05-14 03:58:57.296 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=138
2026-05-14T03:58:57.363851160Z 2026-05-14 03:58:57.363 INFO (qtp985324122-97-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-05-14T03:59:00.773650676Z 2026-05-14 03:59:00.772 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-14T03:59:00.901867149Z 2026-05-14 03:59:00.901 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-14T03:59:05.132383656Z 2026-05-14 03:59:05.131 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-05-14T03:59:05.159289342Z 2026-05-14 03:59:05.158 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-05-14T03:59:05.186129234Z 2026-05-14 03:59:05.185 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-14T03:59:05.466691142Z 2026-05-14 03:59:05.465 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-14T03:59:05.481844746Z 2026-05-14 03:59:05.481 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-14T03:59:05.487416836Z 2026-05-14 03:59:05.486 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:05.532519327Z 2026-05-14 03:59:05.531 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-14T03:59:05.532545612Z "core":"test-collection_shard3_replica_n1",
2026-05-14T03:59:05.532549289Z "node_name":"solr-node1:8983_solr",
2026-05-14T03:59:05.532551952Z "base_url":"http://solr-node1:8983/solr",
2026-05-14T03:59:05.532554561Z "collection":"test-collection",
2026-05-14T03:59:05.532557249Z "shard":"shard3",
2026-05-14T03:59:05.532559752Z "state":"down",
2026-05-14T03:59:05.532562276Z "type":"NRT",
2026-05-14T03:59:05.532564790Z "operation":"ADDREPLICA",
2026-05-14T03:59:05.532567330Z "waitForFinalState":"false"}
2026-05-14T03:59:05.551053786Z 2026-05-14 03:59:05.547 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-14T03:59:05.551075935Z "core":"test-collection_shard2_replica_n2",
2026-05-14T03:59:05.551079163Z "node_name":"solr-node3:8985_solr",
2026-05-14T03:59:05.551081921Z "base_url":"http://solr-node3:8985/solr",
2026-05-14T03:59:05.551084433Z "collection":"test-collection",
2026-05-14T03:59:05.551087039Z "shard":"shard2",
2026-05-14T03:59:05.551089435Z "state":"down",
2026-05-14T03:59:05.551091926Z "type":"NRT",
2026-05-14T03:59:05.551094324Z "operation":"ADDREPLICA",
2026-05-14T03:59:05.551096767Z "waitForFinalState":"false"}
2026-05-14T03:59:05.559382641Z 2026-05-14 03:59:05.556 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-14T03:59:05.559402678Z "core":"test-collection_shard1_replica_n4",
2026-05-14T03:59:05.559405924Z "node_name":"solr-node2:8984_solr",
2026-05-14T03:59:05.559420416Z "base_url":"http://solr-node2:8984/solr",
2026-05-14T03:59:05.559422502Z "collection":"test-collection",
2026-05-14T03:59:05.559424402Z "shard":"shard1",
2026-05-14T03:59:05.559426286Z "state":"down",
2026-05-14T03:59:05.559428213Z "type":"NRT",
2026-05-14T03:59:05.559430163Z "operation":"ADDREPLICA",
2026-05-14T03:59:05.559432092Z "waitForFinalState":"false"}
2026-05-14T03:59:05.673572161Z 2026-05-14 03:59:05.672 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:05.805398475Z 2026-05-14 03:59:05.804 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-14T03:59:05.870185973Z 2026-05-14 03:59:05.869 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-14T03:59:05.933255968Z 2026-05-14 03:59:05.932 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-14T03:59:06.314755381Z 2026-05-14 03:59:06.314 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-14T03:59:06.314852410Z 2026-05-14 03:59:06.314 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-14T03:59:06.340912556Z 2026-05-14 03:59:06.340 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-14T03:59:07.239268718Z 2026-05-14 03:59:07.237 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-14T03:59:07.286389304Z 2026-05-14 03:59:07.285 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-14T03:59:07.297650445Z 2026-05-14 03:59:07.297 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-14T03:59:07.299726615Z 2026-05-14 03:59:07.298 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-14T03:59:07.299822303Z 2026-05-14 03:59:07.299 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-14T03:59:07.382731565Z 2026-05-14 03:59:07.382 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-14T03:59:07.386339373Z 2026-05-14 03:59:07.385 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-14T03:59:07.388291920Z 2026-05-14 03:59:07.387 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-14T03:59:07.389765969Z 2026-05-14 03:59:07.389 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-14T03:59:07.404047565Z 2026-05-14 03:59:07.403 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-14T03:59:07.417302943Z 2026-05-14 03:59:07.416 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-14T03:59:07.428750930Z 2026-05-14 03:59:07.428 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-14T03:59:07.428786621Z 2026-05-14 03:59:07.428 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-14T03:59:07.443569826Z 2026-05-14 03:59:07.442 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-14T03:59:07.445238007Z 2026-05-14 03:59:07.444 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-14T03:59:07.449335883Z 2026-05-14 03:59:07.449 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-14T03:59:07.459895326Z 2026-05-14 03:59:07.459 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-14T03:59:07.459918610Z 2026-05-14 03:59:07.459 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-14T03:59:07.459921975Z 2026-05-14 03:59:07.459 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-14T03:59:07.461752618Z 2026-05-14 03:59:07.461 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-14T03:59:07.462268781Z 2026-05-14 03:59:07.462 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-14T03:59:07.465491326Z 2026-05-14 03:59:07.465 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72130694953107456-core_node3-n_0000000000
2026-05-14T03:59:07.471350709Z 2026-05-14 03:59:07.471 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-14T03:59:07.577413000Z 2026-05-14 03:59:07.576 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:07.582933708Z 2026-05-14 03:59:07.582 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-14T03:59:07.591684567Z 2026-05-14 03:59:07.591 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1790
2026-05-14T03:59:07.690288817Z 2026-05-14 03:59:07.689 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1144308,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-14T03:58:52.281986301Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-14T03:58:57.663046158Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-14T03:58:57.663072603Z It should be set to 65000 to avoid operational disruption.
2026-05-14T03:58:57.663075675Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-14T03:58:57.663357177Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-14T03:58:57.663361038Z
2026-05-14T03:58:57.672714484Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-14T03:58:57.680730801Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-14T03:58:57.722660961Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-14T03:58:57.753609234Z WARNING: A command line option has enabled the Security Manager
2026-05-14T03:58:57.753626314Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-14T03:58:58.937812620Z 2026-05-14 03:58:58.934 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-14T03:58:59.215473566Z 2026-05-14 03:58:59.214 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-14T03:58:59.240008410Z 2026-05-14 03:58:59.239 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-14T03:58:59.246867919Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-14T03:58:59.246898592Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-14T03:58:59.247023152Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-14T03:58:59.247199007Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-14T03:58:59.246490413Z
2026-05-14T03:58:59.249575262Z 2026-05-14 03:58:59.249 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-14T03:58:59.270734725Z 2026-05-14 03:58:59.270 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-14T03:58:59.293893426Z 2026-05-14 03:58:59.293 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-14T03:58:59.436852358Z 2026-05-14 03:58:59.436 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-14T03:58:59.455275618Z 2026-05-14 03:58:59.454 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-14T03:58:59.455440808Z 2026-05-14 03:58:59.454 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-14T03:58:59.455570771Z 2026-05-14 03:58:59.455 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-14T03:58:59.565974431Z 2026-05-14 03:58:59.565 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-14T03:58:59.566114392Z 2026-05-14 03:58:59.565 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-14T03:58:59.566251022Z 2026-05-14 03:58:59.565 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-14T03:58:59.605195612Z 2026-05-14 03:58:59.604 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-14T03:59:00.455697248Z 2026-05-14 03:59:00.455 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-14T03:59:00.613900062Z 2026-05-14 03:59:00.613 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-14T03:59:00.615223544Z 2026-05-14 03:59:00.613 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-14T03:59:00.624357395Z 2026-05-14 03:59:00.623 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-14T03:59:00.629593566Z 2026-05-14 03:59:00.629 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-14T03:59:00.633474073Z 2026-05-14 03:59:00.633 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-14T03:59:00.633564309Z 2026-05-14 03:59:00.633 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-14T03:59:00.667856564Z 2026-05-14 03:59:00.667 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-14T03:59:00.707931564Z 2026-05-14 03:59:00.706 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-14T03:59:00.738078283Z 2026-05-14 03:59:00.737 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-14T03:59:00.750287424Z 2026-05-14 03:59:00.749 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-14T03:59:00.761687616Z 2026-05-14 03:59:00.761 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-14T03:59:00.774708614Z 2026-05-14 03:59:00.774 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-14T03:59:00.818238563Z 2026-05-14 03:59:00.817 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-14T03:59:00.905193040Z 2026-05-14 03:59:00.904 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-14T03:59:01.173838006Z 2026-05-14 03:59:01.173 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-14T03:59:01.431267727Z 2026-05-14 03:59:01.430 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-14T03:59:01.772111607Z 2026-05-14 03:59:01.770 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-14T03:59:01.956932857Z 2026-05-14 03:59:01.956 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-14T03:59:01.966186996Z 2026-05-14 03:59:01.965 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_14.request.log
2026-05-14T03:59:01.970888066Z 2026-05-14 03:59:01.970 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-14T03:59:01.973275011Z 2026-05-14 03:59:01.972 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4293ms
2026-05-14T03:59:02.242850000Z 2026-05-14 03:59:02.241 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=50
2026-05-14T03:59:02.302825473Z 2026-05-14 03:59:02.302 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-05-14T03:59:05.796904644Z 2026-05-14 03:59:05.796 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-14T03:59:05.853438638Z 2026-05-14 03:59:05.852 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-14T03:59:05.932490996Z 2026-05-14 03:59:05.931 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-14T03:59:06.252588796Z 2026-05-14 03:59:06.252 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-14T03:59:06.252612757Z 2026-05-14 03:59:06.252 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-14T03:59:06.272293458Z 2026-05-14 03:59:06.271 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-14T03:59:06.710306260Z 2026-05-14 03:59:06.709 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-14T03:59:06.733480895Z 2026-05-14 03:59:06.732 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-14T03:59:06.741139343Z 2026-05-14 03:59:06.740 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-14T03:59:06.742508861Z 2026-05-14 03:59:06.741 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-14T03:59:06.742525269Z 2026-05-14 03:59:06.742 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-14T03:59:06.789602393Z 2026-05-14 03:59:06.789 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-14T03:59:06.793397457Z 2026-05-14 03:59:06.792 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-14T03:59:06.795648537Z 2026-05-14 03:59:06.795 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-14T03:59:06.797449060Z 2026-05-14 03:59:06.797 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-14T03:59:06.811537755Z 2026-05-14 03:59:06.810 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-14T03:59:06.820661221Z 2026-05-14 03:59:06.820 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-14T03:59:06.830334920Z 2026-05-14 03:59:06.829 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-14T03:59:06.830360884Z 2026-05-14 03:59:06.829 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-14T03:59:06.838239664Z 2026-05-14 03:59:06.837 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-14T03:59:06.850490893Z 2026-05-14 03:59:06.849 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-14T03:59:06.852175272Z 2026-05-14 03:59:06.851 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-14T03:59:06.875630388Z 2026-05-14 03:59:06.875 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-14T03:59:06.875729038Z 2026-05-14 03:59:06.875 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-14T03:59:06.875748998Z 2026-05-14 03:59:06.875 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-14T03:59:06.877235155Z 2026-05-14 03:59:06.876 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-14T03:59:06.878069788Z 2026-05-14 03:59:06.877 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-14T03:59:06.879395487Z 2026-05-14 03:59:06.879 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72130694953107459-core_node6-n_0000000000
2026-05-14T03:59:06.890521427Z 2026-05-14 03:59:06.890 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-14T03:59:07.179875231Z 2026-05-14 03:59:07.179 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:07.184605447Z 2026-05-14 03:59:07.184 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-14T03:59:07.195095379Z 2026-05-14 03:59:07.194 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1399
2026-05-14T03:59:07.364093474Z 2026-05-14 03:59:07.363 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:07.577765148Z 2026-05-14 03:59:07.577 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:07.690689459Z 2026-05-14 03:59:07.690 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1144703,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-14T03:58:57.452471801Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-14T03:58:57.890464787Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-14T03:58:57.890504791Z It should be set to 65000 to avoid operational disruption.
2026-05-14T03:58:57.890508388Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-14T03:58:57.890773289Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-14T03:58:57.890801572Z
2026-05-14T03:58:57.900536964Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-14T03:58:57.908953304Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-14T03:58:57.957767606Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-14T03:58:57.989945337Z WARNING: A command line option has enabled the Security Manager
2026-05-14T03:58:57.989970091Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-14T03:58:59.148239891Z 2026-05-14 03:58:59.144 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-14T03:58:59.417848969Z 2026-05-14 03:58:59.417 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-14T03:58:59.444020698Z 2026-05-14 03:58:59.443 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-14T03:58:59.451563297Z 2026-05-14 03:58:59.450 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-14T03:58:59.451585500Z 2026-05-14 03:58:59.451 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-14T03:58:59.451706926Z 2026-05-14 03:58:59.451 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-14T03:58:59.451844704Z 2026-05-14 03:58:59.451 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-14T03:58:59.451249098Z
2026-05-14T03:58:59.454754900Z 2026-05-14 03:58:59.454 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-14T03:58:59.477280340Z 2026-05-14 03:58:59.476 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-14T03:58:59.501312262Z 2026-05-14 03:58:59.500 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-14T03:58:59.646816369Z 2026-05-14 03:58:59.646 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-14T03:58:59.666792371Z 2026-05-14 03:58:59.666 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-14T03:58:59.666864765Z 2026-05-14 03:58:59.666 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-14T03:58:59.666996413Z 2026-05-14 03:58:59.666 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-14T03:58:59.776253466Z 2026-05-14 03:58:59.775 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-14T03:58:59.776561903Z 2026-05-14 03:58:59.776 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-14T03:58:59.776693015Z 2026-05-14 03:58:59.776 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-14T03:58:59.812192357Z 2026-05-14 03:58:59.811 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-14T03:59:00.647394937Z 2026-05-14 03:59:00.646 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-14T03:59:00.783123490Z 2026-05-14 03:59:00.782 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-14T03:59:00.784457380Z 2026-05-14 03:59:00.782 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-14T03:59:00.793454234Z 2026-05-14 03:59:00.793 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-14T03:59:00.798514987Z 2026-05-14 03:59:00.798 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-14T03:59:00.801873112Z 2026-05-14 03:59:00.801 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-14T03:59:00.802127529Z 2026-05-14 03:59:00.801 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-14T03:59:00.834871005Z 2026-05-14 03:59:00.834 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-14T03:59:00.844605617Z 2026-05-14 03:59:00.844 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-14T03:59:00.865676694Z 2026-05-14 03:59:00.865 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-14T03:59:00.878802944Z 2026-05-14 03:59:00.878 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-14T03:59:00.891674749Z 2026-05-14 03:59:00.891 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-14T03:59:00.903902935Z 2026-05-14 03:59:00.903 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-14T03:59:00.943988145Z 2026-05-14 03:59:00.943 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-14T03:59:01.228763534Z 2026-05-14 03:59:01.228 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-14T03:59:01.420570885Z 2026-05-14 03:59:01.419 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-14T03:59:01.728369034Z 2026-05-14 03:59:01.727 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-14T03:59:01.871283804Z 2026-05-14 03:59:01.870 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-14T03:59:01.878148635Z 2026-05-14 03:59:01.877 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_14.request.log
2026-05-14T03:59:01.882065836Z 2026-05-14 03:59:01.881 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-14T03:59:01.883154553Z 2026-05-14 03:59:01.882 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3975ms
2026-05-14T03:59:03.974888370Z 2026-05-14 03:59:03.969 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=67
2026-05-14T03:59:04.042766822Z 2026-05-14 03:59:04.041 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=22
2026-05-14T03:59:05.431165535Z 2026-05-14 03:59:05.430 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=9
2026-05-14T03:59:05.796910967Z 2026-05-14 03:59:05.796 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-14T03:59:05.857066010Z 2026-05-14 03:59:05.856 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-14T03:59:05.927027548Z 2026-05-14 03:59:05.926 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-14T03:59:06.392167340Z 2026-05-14 03:59:06.391 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-14T03:59:06.392384945Z 2026-05-14 03:59:06.391 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-14T03:59:06.410370259Z 2026-05-14 03:59:06.409 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-14T03:59:06.865434296Z 2026-05-14 03:59:06.864 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-14T03:59:06.889861081Z 2026-05-14 03:59:06.889 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-14T03:59:06.899182839Z 2026-05-14 03:59:06.898 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-14T03:59:06.900259017Z 2026-05-14 03:59:06.899 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-14T03:59:06.900345354Z 2026-05-14 03:59:06.900 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-14T03:59:06.948168790Z 2026-05-14 03:59:06.947 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-14T03:59:06.951564003Z 2026-05-14 03:59:06.951 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-14T03:59:07.178102926Z 2026-05-14 03:59:07.176 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-14T03:59:07.181085263Z 2026-05-14 03:59:07.180 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-14T03:59:07.196040210Z 2026-05-14 03:59:07.195 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-14T03:59:07.205571992Z 2026-05-14 03:59:07.204 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-14T03:59:07.213411388Z 2026-05-14 03:59:07.212 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-14T03:59:07.213456655Z 2026-05-14 03:59:07.213 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-14T03:59:07.220350349Z 2026-05-14 03:59:07.219 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-14T03:59:07.228691484Z 2026-05-14 03:59:07.228 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-14T03:59:07.230231066Z 2026-05-14 03:59:07.229 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-14T03:59:07.242980867Z 2026-05-14 03:59:07.242 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-14T03:59:07.243003214Z 2026-05-14 03:59:07.242 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-14T03:59:07.243059336Z 2026-05-14 03:59:07.242 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-14T03:59:07.244457825Z 2026-05-14 03:59:07.244 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-14T03:59:07.245117314Z 2026-05-14 03:59:07.244 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-14T03:59:07.246374648Z 2026-05-14 03:59:07.246 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72130694953107460-core_node5-n_0000000000
2026-05-14T03:59:07.256887314Z 2026-05-14 03:59:07.256 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-14T03:59:07.364285761Z 2026-05-14 03:59:07.363 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:07.368562322Z 2026-05-14 03:59:07.368 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-14T03:59:07.382341669Z 2026-05-14 03:59:07.381 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1587
2026-05-14T03:59:07.577822128Z 2026-05-14 03:59:07.577 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:07.603827937Z 2026-05-14 03:59:07.603 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-14T03:59:07.690657707Z 2026-05-14 03:59:07.690 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-14T03:59:07.693407698Z 2026-05-14 03:59:07.693 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2248
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1144889,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-14T03:58:57.679867425Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 26755 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# 87f3bb8db2f4: Already exists
# 7c0b62ebdf15: Already exists
# 5b95479e5f9b: Already exists
# c913227527c9: Already exists
# a34ba287db10: Pulling fs layer
# d1122302140a: Pulling fs layer
# e63de72c09be: Pulling fs layer
# ddd33485f40b: Pulling fs layer
# 67aedfcd29ab: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# ddd33485f40b: Waiting
# 67aedfcd29ab: Waiting
# 4f4fb700ef54: Waiting
# d1122302140a: Verifying Checksum
# d1122302140a: Download complete
# e63de72c09be: Download complete
# ddd33485f40b: Download complete
# 4f4fb700ef54: Download complete
# 67aedfcd29ab: Verifying Checksum
# 67aedfcd29ab: Download complete
# a34ba287db10: Verifying Checksum
# a34ba287db10: Download complete
# a34ba287db10: Pull complete
# d1122302140a: Pull complete
# e63de72c09be: Pull complete
# ddd33485f40b: Pull complete
# 67aedfcd29ab: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:ade140d83779bd4ea80626422a06cc925e99b05dd5c0df36ef88ceae30d3b74a
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 5de6877746e1717837de99a07fbb11933201e4d11914e49878095ea74796c910
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 27821a8d01463d225b1b94a437f2b59def0356d9b6d91ad0b81d4a4ff7d77dea
# Starting solr-node2 connected to first node's ZooKeeper
# c16a39090f53bd3a261d9ea3ade361682ed9687e5afb8096a2d8cb7254ad8a9d
# Started solr-node3 connected to first node's ZooKeeper
# 5c76b1370d53d7cae07e72280f59f6699d1a4a049c4f0b2cc8a60e0abfb81c6d
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-14 03:59:05.225; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-14 03:59:05.257; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 16409 ms
ok 85 snapshot lifecycle # in 4845 ms
ok 86 snapshot list # in 3355 ms
ok 87 snapshot describe # in 3349 ms
ok 88 start solr with ssl # in 16914 ms
ok 89 use different hostname when not checking peer-name # in 25410 ms
ok 90 start solr with ssl and auth # in 14784 ms
ok 91 start solr with client truststore and security manager # in 16947 ms
ok 92 start solr with mTLS needed # in 47502 ms
ok 93 start solr with mTLS wanted # in 36514 ms
ok 94 test keystore reload # in 60697 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 16188 ms
ok 96 stop command for single port # in 18727 ms
ok 97 check stop command doesn't hang # in 224 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8895 ms
ok 99 deprecated system properties converted to modern properties # in 195316 ms
ok 100 start with custom jetty options # in 8305 ms
ok 101 bootstrapping a configset # in 9448 ms
ok 102 status detects locally running solr # in 9654 ms
ok 103 status with --solr-url from user # in 8456 ms
ok 104 status with --port from user # in 8650 ms
ok 105 multiple connection options are prevented # in 881 ms
ok 106 status with invalid --solr-url from user # in 1286 ms
ok 107 status with --short format # in 196108 ms
ok 108 searching solr via locally executed streaming expression # in 1748 ms
ok 109 searching solr via remotely executed streaming expression # in 1307 ms
ok 110 variable interpolation # in 1698 ms
ok 111 searching solr without credentials fails with error # in 1152 ms
ok 112 --version returns Solr version # in 746 ms
ok 113 version as direct tool call still runs # in 752 ms
ok 114 short help # in 860 ms
ok 115 short help is inferred # in 792 ms
ok 116 long help # in 847 ms
ok 117 running subcommands with zk is prevented # in 733 ms
ok 118 listing out files # in 2184 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 8255 ms
ok 120 copying files around # in 11170 ms
ok 121 upconfig # in 2680 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1239 ms
ok 123 downconfig # in 1214 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1497 ms
ok 125 env var ZK_HOST is honored # in 3312 ms
ok 126 Cleanup # in 1890 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 30m 13s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/4yaimmavwenku
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 207 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/207/
No tests ran.
Build Log:
[...truncated 433 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:39281/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:39281/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:39281 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-12 12:35:49.819; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-12 12:35:49.822; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:39281/solr/admin/collections
not ok 81 verbose echo the Solr response # in 1133 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281.
#
# ERROR: Server refused connection at: http://localhost:39281/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281.
#
# ERROR: Server refused connection at: http://localhost:39281/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8546 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 16 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-12T12:36:18.108407718Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-12T12:36:18.108460959Z It should be set to 65000 to avoid operational disruption.
2026-05-12T12:36:18.108464809Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-12T12:36:18.108785180Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-12T12:36:18.108797263Z
2026-05-12T12:36:18.118890777Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-12T12:36:18.128399232Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-12T12:36:18.177363477Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-12T12:36:18.213141502Z WARNING: A command line option has enabled the Security Manager
2026-05-12T12:36:18.213174625Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-12T12:36:19.453575062Z 2026-05-12 12:36:19.449 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-12T12:36:19.736179385Z 2026-05-12 12:36:19.735 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-12T12:36:19.764142387Z 2026-05-12 12:36:19.763 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-12T12:36:19.771825704Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-12T12:36:19.772025343Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-12T12:36:19.772205538Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-12T12:36:19.772410250Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-12T12:36:19.771430816Z
2026-05-12T12:36:19.774908300Z 2026-05-12 12:36:19.774 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-12T12:36:19.797465170Z 2026-05-12 12:36:19.796 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-12T12:36:19.823970621Z 2026-05-12 12:36:19.823 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-12T12:36:19.823999137Z 2026-05-12 12:36:19.823 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-12T12:36:19.863679664Z 2026-05-12 12:36:19.862 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-12T12:36:20.901301838Z 2026-05-12 12:36:20.900 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-12T12:36:21.227682836Z 2026-05-12 12:36:21.226 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-12T12:36:21.234386858Z 2026-05-12 12:36:21.233 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-12T12:36:21.234529192Z 2026-05-12 12:36:21.233 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-12T12:36:21.238628296Z 2026-05-12 12:36:21.238 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-12T12:36:21.238719690Z 2026-05-12 12:36:21.238 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-12T12:36:21.311739936Z 2026-05-12 12:36:21.311 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-12T12:36:21.741465614Z 2026-05-12 12:36:21.740 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-12T12:36:21.754015857Z 2026-05-12 12:36:21.753 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-12T12:36:21.839683110Z 2026-05-12 12:36:21.838 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-12T12:36:21.861759991Z 2026-05-12 12:36:21.861 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-12T12:36:21.861895044Z 2026-05-12 12:36:21.861 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-12T12:36:21.987536224Z 2026-05-12 12:36:21.987 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-12T12:36:22.011542156Z 2026-05-12 12:36:22.011 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-12T12:36:22.028150035Z 2026-05-12 12:36:22.027 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-12T12:36:22.038259406Z 2026-05-12 12:36:22.037 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72121319242924032-solr-node1:8983_solr-n_0000000000) starting
2026-05-12T12:36:22.103526695Z 2026-05-12 12:36:22.102 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-12T12:36:22.103618455Z 2026-05-12 12:36:22.103 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-12T12:36:22.112606041Z 2026-05-12 12:36:22.112 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-12T12:36:22.150367884Z 2026-05-12 12:36:22.149 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-12T12:36:22.573225895Z 2026-05-12 12:36:22.571 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-12T12:36:22.734980585Z 2026-05-12 12:36:22.734 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-12T12:36:22.980314954Z 2026-05-12 12:36:22.979 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-12T12:36:23.104035652Z 2026-05-12 12:36:23.103 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-12T12:36:23.109442883Z 2026-05-12 12:36:23.109 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_12.request.log
2026-05-12T12:36:23.145617495Z 2026-05-12 12:36:23.113 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-12T12:36:23.147416717Z 2026-05-12 12:36:23.147 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5020ms
2026-05-12T12:36:23.340377012Z 2026-05-12 12:36:23.339 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=63
2026-05-12T12:36:23.397355505Z 2026-05-12 12:36:23.396 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-12T12:36:26.576112169Z 2026-05-12 12:36:26.575 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-12T12:36:26.914300419Z 2026-05-12 12:36:26.913 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-12T12:36:30.914073714Z 2026-05-12 12:36:30.913 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=36
2026-05-12T12:36:30.939800643Z 2026-05-12 12:36:30.939 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=10
2026-05-12T12:36:30.956902933Z 2026-05-12 12:36:30.956 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-12T12:36:31.332529879Z 2026-05-12 12:36:31.331 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-12T12:36:31.371939271Z 2026-05-12 12:36:31.371 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-12T12:36:31.400301079Z 2026-05-12 12:36:31.399 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:31.475919003Z 2026-05-12 12:36:31.475 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-12T12:36:31.475941424Z "core":"test-collection_shard1_replica_n1",
2026-05-12T12:36:31.475944632Z "node_name":"solr-node2:8984_solr",
2026-05-12T12:36:31.475947269Z "base_url":"http://solr-node2:8984/solr",
2026-05-12T12:36:31.475950275Z "collection":"test-collection",
2026-05-12T12:36:31.475952892Z "shard":"shard1",
2026-05-12T12:36:31.475955394Z "state":"down",
2026-05-12T12:36:31.475958016Z "type":"NRT",
2026-05-12T12:36:31.475960439Z "operation":"ADDREPLICA",
2026-05-12T12:36:31.475963016Z "waitForFinalState":"false"}
2026-05-12T12:36:31.488029416Z 2026-05-12 12:36:31.487 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-12T12:36:31.488051374Z "core":"test-collection_shard2_replica_n3",
2026-05-12T12:36:31.488054930Z "node_name":"solr-node3:8985_solr",
2026-05-12T12:36:31.488057989Z "base_url":"http://solr-node3:8985/solr",
2026-05-12T12:36:31.488060838Z "collection":"test-collection",
2026-05-12T12:36:31.488063313Z "shard":"shard2",
2026-05-12T12:36:31.488065787Z "state":"down",
2026-05-12T12:36:31.488068274Z "type":"NRT",
2026-05-12T12:36:31.488070652Z "operation":"ADDREPLICA",
2026-05-12T12:36:31.488073083Z "waitForFinalState":"false"}
2026-05-12T12:36:31.492181156Z 2026-05-12 12:36:31.491 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-12T12:36:31.492197999Z "core":"test-collection_shard3_replica_n4",
2026-05-12T12:36:31.492201068Z "node_name":"solr-node1:8983_solr",
2026-05-12T12:36:31.492214122Z "base_url":"http://solr-node1:8983/solr",
2026-05-12T12:36:31.492216279Z "collection":"test-collection",
2026-05-12T12:36:31.492218222Z "shard":"shard3",
2026-05-12T12:36:31.492220100Z "state":"down",
2026-05-12T12:36:31.492222065Z "type":"NRT",
2026-05-12T12:36:31.492223993Z "operation":"ADDREPLICA",
2026-05-12T12:36:31.492225896Z "waitForFinalState":"false"}
2026-05-12T12:36:31.606620249Z 2026-05-12 12:36:31.606 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:31.744648544Z 2026-05-12 12:36:31.744 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-12T12:36:31.864612087Z 2026-05-12 12:36:31.864 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-12T12:36:31.962501729Z 2026-05-12 12:36:31.961 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-12T12:36:32.517215377Z 2026-05-12 12:36:32.516 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-12T12:36:32.517240403Z 2026-05-12 12:36:32.516 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-12T12:36:32.554565925Z 2026-05-12 12:36:32.554 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n4], dataDir=[/var/solr/data/test-collection_shard3_replica_n4/data/]
2026-05-12T12:36:34.065270171Z 2026-05-12 12:36:34.064 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-12T12:36:34.128986710Z 2026-05-12 12:36:34.128 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-12T12:36:34.140046876Z 2026-05-12 12:36:34.139 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-12T12:36:34.141041822Z 2026-05-12 12:36:34.140 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-12T12:36:34.141120674Z 2026-05-12 12:36:34.140 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-12T12:36:34.254091526Z 2026-05-12 12:36:34.253 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-12T12:36:34.258448961Z 2026-05-12 12:36:34.258 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-12T12:36:34.261489058Z 2026-05-12 12:36:34.261 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-12T12:36:34.263254309Z 2026-05-12 12:36:34.263 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-12T12:36:34.284533250Z 2026-05-12 12:36:34.284 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-12T12:36:34.292608168Z 2026-05-12 12:36:34.292 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-12T12:36:34.296896735Z 2026-05-12 12:36:34.296 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-12T12:36:34.296998001Z 2026-05-12 12:36:34.296 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-12T12:36:34.308953882Z 2026-05-12 12:36:34.308 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-12T12:36:34.315752978Z 2026-05-12 12:36:34.315 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-12T12:36:34.316996841Z 2026-05-12 12:36:34.316 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-12T12:36:34.335370097Z 2026-05-12 12:36:34.335 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-12T12:36:34.335389875Z 2026-05-12 12:36:34.335 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-12T12:36:34.335462531Z 2026-05-12 12:36:34.335 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n4/
2026-05-12T12:36:34.336373071Z 2026-05-12 12:36:34.336 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-12T12:36:34.336796168Z 2026-05-12 12:36:34.336 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ has no replicas
2026-05-12T12:36:34.337706242Z 2026-05-12 12:36:34.337 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72121319242924032-core_node6-n_0000000000
2026-05-12T12:36:34.342478826Z 2026-05-12 12:36:34.342 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ shard3
2026-05-12T12:36:34.447654165Z 2026-05-12 12:36:34.447 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:34.465192025Z 2026-05-12 12:36:34.464 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-12T12:36:34.471878160Z 2026-05-12 12:36:34.471 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2730
2026-05-12T12:36:34.571583497Z 2026-05-12 12:36:34.571 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1699922,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-12T12:36:17.876237366Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-12T12:36:23.670757896Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-12T12:36:23.670785354Z It should be set to 65000 to avoid operational disruption.
2026-05-12T12:36:23.670788579Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-12T12:36:23.671106032Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-12T12:36:23.671121907Z
2026-05-12T12:36:23.679783264Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-12T12:36:23.687178999Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-12T12:36:23.730390851Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-12T12:36:23.763789395Z WARNING: A command line option has enabled the Security Manager
2026-05-12T12:36:23.763810673Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-12T12:36:24.810398178Z 2026-05-12 12:36:24.807 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-12T12:36:25.073214439Z 2026-05-12 12:36:25.072 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-12T12:36:25.097751282Z 2026-05-12 12:36:25.097 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-12T12:36:25.104878875Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-12T12:36:25.104900681Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-12T12:36:25.105072145Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-12T12:36:25.105221114Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-12T12:36:25.104494308Z
2026-05-12T12:36:25.107663101Z 2026-05-12 12:36:25.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-12T12:36:25.129828719Z 2026-05-12 12:36:25.129 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-12T12:36:25.154787827Z 2026-05-12 12:36:25.154 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-12T12:36:25.289638762Z 2026-05-12 12:36:25.289 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-12T12:36:25.306143344Z 2026-05-12 12:36:25.305 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-12T12:36:25.306203858Z 2026-05-12 12:36:25.305 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-12T12:36:25.306328436Z 2026-05-12 12:36:25.305 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-12T12:36:25.416130470Z 2026-05-12 12:36:25.415 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-12T12:36:25.416148135Z 2026-05-12 12:36:25.415 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-12T12:36:25.416211060Z 2026-05-12 12:36:25.415 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-12T12:36:25.448794830Z 2026-05-12 12:36:25.448 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-12T12:36:26.261644186Z 2026-05-12 12:36:26.260 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-12T12:36:26.414033022Z 2026-05-12 12:36:26.411 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-12T12:36:26.414061064Z 2026-05-12 12:36:26.412 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-12T12:36:26.423710030Z 2026-05-12 12:36:26.423 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-12T12:36:26.428886466Z 2026-05-12 12:36:26.428 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-12T12:36:26.434270270Z 2026-05-12 12:36:26.433 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-12T12:36:26.434288310Z 2026-05-12 12:36:26.433 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-12T12:36:26.481539403Z 2026-05-12 12:36:26.480 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-12T12:36:26.490999626Z 2026-05-12 12:36:26.490 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-12T12:36:26.514551793Z 2026-05-12 12:36:26.513 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-12T12:36:26.549680420Z 2026-05-12 12:36:26.548 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-12T12:36:26.562182362Z 2026-05-12 12:36:26.561 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-12T12:36:26.577302606Z 2026-05-12 12:36:26.576 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-12T12:36:26.627496562Z 2026-05-12 12:36:26.626 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-12T12:36:26.915124349Z 2026-05-12 12:36:26.913 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-12T12:36:26.957674800Z 2026-05-12 12:36:26.957 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-12T12:36:27.156005864Z 2026-05-12 12:36:27.155 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-12T12:36:27.495654640Z 2026-05-12 12:36:27.494 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-12T12:36:27.657563934Z 2026-05-12 12:36:27.656 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-12T12:36:27.664674752Z 2026-05-12 12:36:27.664 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_12.request.log
2026-05-12T12:36:27.670156840Z 2026-05-12 12:36:27.668 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-12T12:36:27.671910269Z 2026-05-12 12:36:27.671 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3985ms
2026-05-12T12:36:27.878717051Z 2026-05-12 12:36:27.878 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=67
2026-05-12T12:36:27.957115637Z 2026-05-12 12:36:27.956 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-05-12T12:36:31.742701263Z 2026-05-12 12:36:31.742 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node2&name=test-collection_shard1_replica_n1&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-12T12:36:31.790912156Z 2026-05-12 12:36:31.790 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-12T12:36:31.870106004Z 2026-05-12 12:36:31.869 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-12T12:36:32.177629854Z 2026-05-12 12:36:32.177 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-12T12:36:32.177657374Z 2026-05-12 12:36:32.177 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-12T12:36:32.195749540Z 2026-05-12 12:36:32.195 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n1], dataDir=[/var/solr/data/test-collection_shard1_replica_n1/data/]
2026-05-12T12:36:32.615671583Z 2026-05-12 12:36:32.615 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-12T12:36:32.633365566Z 2026-05-12 12:36:32.632 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-12T12:36:32.636618509Z 2026-05-12 12:36:32.636 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-12T12:36:32.637717232Z 2026-05-12 12:36:32.637 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-12T12:36:32.637731005Z 2026-05-12 12:36:32.637 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-12T12:36:32.674722513Z 2026-05-12 12:36:32.674 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-12T12:36:32.677271755Z 2026-05-12 12:36:32.676 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-12T12:36:32.678897080Z 2026-05-12 12:36:32.678 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-12T12:36:32.680452063Z 2026-05-12 12:36:32.680 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-12T12:36:32.691239539Z 2026-05-12 12:36:32.690 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-12T12:36:32.698908919Z 2026-05-12 12:36:32.698 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-12T12:36:32.703785268Z 2026-05-12 12:36:32.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node2 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-12T12:36:32.703799978Z 2026-05-12 12:36:32.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node2 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-12T12:36:32.710720821Z 2026-05-12 12:36:32.710 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node2 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-12T12:36:32.723147803Z 2026-05-12 12:36:32.722 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node2=0}, version=0} for registerTerm
2026-05-12T12:36:32.724551622Z 2026-05-12 12:36:32.724 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-12T12:36:32.739181333Z 2026-05-12 12:36:32.738 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-12T12:36:32.739200291Z 2026-05-12 12:36:32.738 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-12T12:36:32.739258401Z 2026-05-12 12:36:32.739 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n1/
2026-05-12T12:36:32.740498426Z 2026-05-12 12:36:32.740 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-12T12:36:32.740962232Z 2026-05-12 12:36:32.740 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n1/ has no replicas
2026-05-12T12:36:32.742016463Z 2026-05-12 12:36:32.741 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72121319242924035-core_node2-n_0000000000
2026-05-12T12:36:32.751431065Z 2026-05-12 12:36:32.751 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n1/ shard1
2026-05-12T12:36:32.858185570Z 2026-05-12 12:36:32.857 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:32.862313141Z 2026-05-12 12:36:32.861 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-12T12:36:32.871494080Z 2026-05-12 12:36:32.871 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1130
2026-05-12T12:36:32.974735723Z 2026-05-12 12:36:32.974 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 209] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:34.448090962Z 2026-05-12 12:36:34.447 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:34.571769447Z 2026-05-12 12:36:34.571 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1700321,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-12T12:36:23.479358828Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-12T12:36:23.871027852Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-12T12:36:23.871063615Z It should be set to 65000 to avoid operational disruption.
2026-05-12T12:36:23.871066681Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-12T12:36:23.871389768Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-12T12:36:23.871410750Z
2026-05-12T12:36:23.880304696Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-12T12:36:23.887440312Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-12T12:36:23.935593917Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-12T12:36:23.964801246Z WARNING: A command line option has enabled the Security Manager
2026-05-12T12:36:23.964824845Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-12T12:36:25.154319612Z 2026-05-12 12:36:25.151 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-12T12:36:25.416072940Z 2026-05-12 12:36:25.415 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-12T12:36:25.440127540Z 2026-05-12 12:36:25.439 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-12T12:36:25.446740551Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-12T12:36:25.446968677Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-12T12:36:25.446978516Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-12T12:36:25.447122290Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-12T12:36:25.446375032Z
2026-05-12T12:36:25.449344238Z 2026-05-12 12:36:25.448 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-12T12:36:25.470385368Z 2026-05-12 12:36:25.469 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-12T12:36:25.493425874Z 2026-05-12 12:36:25.492 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-12T12:36:25.626933533Z 2026-05-12 12:36:25.626 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-12T12:36:25.643330286Z 2026-05-12 12:36:25.642 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-12T12:36:25.643430229Z 2026-05-12 12:36:25.643 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-12T12:36:25.643577573Z 2026-05-12 12:36:25.643 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-12T12:36:25.753075747Z 2026-05-12 12:36:25.752 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-12T12:36:25.753111945Z 2026-05-12 12:36:25.752 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-12T12:36:25.753245898Z 2026-05-12 12:36:25.752 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-12T12:36:25.792519167Z 2026-05-12 12:36:25.791 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-12T12:36:26.612309676Z 2026-05-12 12:36:26.611 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-12T12:36:26.764358825Z 2026-05-12 12:36:26.763 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-12T12:36:26.766238694Z 2026-05-12 12:36:26.763 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-12T12:36:26.776183731Z 2026-05-12 12:36:26.775 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-12T12:36:26.783087023Z 2026-05-12 12:36:26.781 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-12T12:36:26.788487649Z 2026-05-12 12:36:26.786 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-12T12:36:26.788541329Z 2026-05-12 12:36:26.787 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-12T12:36:26.837665460Z 2026-05-12 12:36:26.836 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-12T12:36:26.848177990Z 2026-05-12 12:36:26.847 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-12T12:36:26.874459526Z 2026-05-12 12:36:26.873 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-12T12:36:26.888605846Z 2026-05-12 12:36:26.887 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-12T12:36:26.901444929Z 2026-05-12 12:36:26.900 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-12T12:36:26.915165390Z 2026-05-12 12:36:26.914 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-12T12:36:26.962292269Z 2026-05-12 12:36:26.961 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-12T12:36:27.274543425Z 2026-05-12 12:36:27.273 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-12T12:36:27.468202655Z 2026-05-12 12:36:27.467 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-12T12:36:27.762629796Z 2026-05-12 12:36:27.761 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-12T12:36:27.892985180Z 2026-05-12 12:36:27.892 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-12T12:36:27.899003210Z 2026-05-12 12:36:27.898 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_12.request.log
2026-05-12T12:36:27.902748828Z 2026-05-12 12:36:27.902 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-12T12:36:27.903636090Z 2026-05-12 12:36:27.903 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4017ms
2026-05-12T12:36:29.762508230Z 2026-05-12 12:36:29.718 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=187
2026-05-12T12:36:29.841461507Z 2026-05-12 12:36:29.840 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=34
2026-05-12T12:36:31.263595054Z 2026-05-12 12:36:31.263 INFO (qtp985324122-59-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=61
2026-05-12T12:36:31.743519840Z 2026-05-12 12:36:31.741 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n3&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-12T12:36:32.017814167Z 2026-05-12 12:36:32.017 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-12T12:36:32.149275961Z 2026-05-12 12:36:32.148 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-12T12:36:32.821134919Z 2026-05-12 12:36:32.820 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-12T12:36:32.821269006Z 2026-05-12 12:36:32.820 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n3' using configuration from configset techproducts, trusted=true
2026-05-12T12:36:32.926194140Z 2026-05-12 12:36:32.925 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n3], dataDir=[/var/solr/data/test-collection_shard2_replica_n3/data/]
2026-05-12T12:36:34.092381438Z 2026-05-12 12:36:34.091 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-12T12:36:34.115750010Z 2026-05-12 12:36:34.115 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-12T12:36:34.118867320Z 2026-05-12 12:36:34.118 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-12T12:36:34.119714417Z 2026-05-12 12:36:34.119 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-12T12:36:34.119770608Z 2026-05-12 12:36:34.119 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-12T12:36:34.174689866Z 2026-05-12 12:36:34.174 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-12T12:36:34.181543300Z 2026-05-12 12:36:34.181 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-12T12:36:34.184241179Z 2026-05-12 12:36:34.183 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-12T12:36:34.187351061Z 2026-05-12 12:36:34.187 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-12T12:36:34.199324745Z 2026-05-12 12:36:34.199 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-12T12:36:34.209013628Z 2026-05-12 12:36:34.208 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-12T12:36:34.216748223Z 2026-05-12 12:36:34.216 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n3 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-12T12:36:34.216820135Z 2026-05-12 12:36:34.216 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n3 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-12T12:36:34.225496538Z 2026-05-12 12:36:34.225 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n3 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-12T12:36:34.258848084Z 2026-05-12 12:36:34.258 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-12T12:36:34.260173373Z 2026-05-12 12:36:34.259 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-12T12:36:34.272071614Z 2026-05-12 12:36:34.271 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-12T12:36:34.272089358Z 2026-05-12 12:36:34.271 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-12T12:36:34.272109385Z 2026-05-12 12:36:34.271 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n3/
2026-05-12T12:36:34.273636893Z 2026-05-12 12:36:34.273 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-12T12:36:34.273802717Z 2026-05-12 12:36:34.273 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n3/ has no replicas
2026-05-12T12:36:34.274826116Z 2026-05-12 12:36:34.274 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72121319242924036-core_node5-n_0000000000
2026-05-12T12:36:34.287304398Z 2026-05-12 12:36:34.287 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n3/ shard2
2026-05-12T12:36:34.448906373Z 2026-05-12 12:36:34.447 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:34.451630642Z 2026-05-12 12:36:34.451 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-12T12:36:34.459123030Z 2026-05-12 12:36:34.458 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n3&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2718
2026-05-12T12:36:34.479539442Z 2026-05-12 12:36:34.479 INFO (qtp985324122-58-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-12T12:36:34.571581427Z 2026-05-12 12:36:34.571 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-12T12:36:34.574114091Z 2026-05-12 12:36:34.573 INFO (qtp985324122-58-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3298
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1700506,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-12T12:36:23.66430058Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 26751 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 3aa9120e46f5: Waiting
# 9cdfe5320de8: Waiting
# 4f4fb700ef54: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 3aa9120e46f5: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# 87f3bb8db2f4: Already exists
# 7c0b62ebdf15: Already exists
# 5b95479e5f9b: Already exists
# c913227527c9: Already exists
# 44d992948799: Pulling fs layer
# 716ed2882a63: Pulling fs layer
# 8372919fc868: Pulling fs layer
# 2198107b532d: Pulling fs layer
# 49f4cafecda6: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 49f4cafecda6: Waiting
# 2198107b532d: Waiting
# 4f4fb700ef54: Waiting
# 8372919fc868: Verifying Checksum
# 8372919fc868: Download complete
# 716ed2882a63: Verifying Checksum
# 716ed2882a63: Download complete
# 49f4cafecda6: Verifying Checksum
# 49f4cafecda6: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 2198107b532d: Download complete
# 44d992948799: Verifying Checksum
# 44d992948799: Download complete
# 44d992948799: Pull complete
# 716ed2882a63: Pull complete
# 8372919fc868: Pull complete
# 2198107b532d: Pull complete
# 49f4cafecda6: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:f0544343d632eb05fdba9371a0f48b98c03354d7394cfdb0e648434d09fc5498
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 7e574ffef0df92be69c0f77afb64fa39b94b1e1e31cfa4f2c22bdca8e55c2763
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 06ba64ef120cc3b6885db0ea762c77450652d827542183dfd09476117e972361
# Starting solr-node2 connected to first node's ZooKeeper
# c9abb6e08d4e65815e07cca5260256e687071ab7350e96fbf8847b1f541787e3
# Started solr-node3 connected to first node's ZooKeeper
# 27e6607e5186255f4113777682fda5d54ad824629f3ff655105c10ff930b9037
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-12 12:36:31.008; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-12 12:36:31.043; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14469 ms
ok 85 snapshot lifecycle # in 4455 ms
ok 86 snapshot list # in 3075 ms
ok 87 snapshot describe # in 3097 ms
ok 88 start solr with ssl # in 15173 ms
ok 89 use different hostname when not checking peer-name # in 23178 ms
ok 90 start solr with ssl and auth # in 13714 ms
ok 91 start solr with client truststore and security manager # in 15319 ms
ok 92 start solr with mTLS needed # in 44460 ms
ok 93 start solr with mTLS wanted # in 34139 ms
ok 94 test keystore reload # in 56067 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15445 ms
ok 96 stop command for single port # in 17631 ms
ok 97 check stop command doesn't hang # in 204 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7742 ms
ok 99 deprecated system properties converted to modern properties # in 195312 ms
ok 100 start with custom jetty options # in 7715 ms
ok 101 bootstrapping a configset # in 8781 ms
ok 102 status detects locally running solr # in 9090 ms
ok 103 status with --solr-url from user # in 7824 ms
ok 104 status with --port from user # in 7696 ms
ok 105 multiple connection options are prevented # in 799 ms
ok 106 status with invalid --solr-url from user # in 1203 ms
ok 107 status with --short format # in 196011 ms
ok 108 searching solr via locally executed streaming expression # in 1643 ms
ok 109 searching solr via remotely executed streaming expression # in 1203 ms
ok 110 variable interpolation # in 1606 ms
ok 111 searching solr without credentials fails with error # in 1092 ms
ok 112 --version returns Solr version # in 688 ms
ok 113 version as direct tool call still runs # in 711 ms
ok 114 short help # in 730 ms
ok 115 short help is inferred # in 725 ms
ok 116 long help # in 734 ms
ok 117 running subcommands with zk is prevented # in 675 ms
ok 118 listing out files # in 2080 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7465 ms
ok 120 copying files around # in 10492 ms
ok 121 upconfig # in 2447 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1053 ms
ok 123 downconfig # in 1100 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1369 ms
ok 125 env var ZK_HOST is honored # in 3091 ms
ok 126 Cleanup # in 1823 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/tg6vr5opsgi4y
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 206 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/206/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-05-11 12:54:35.959; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-11 12:54:35.962; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-11 12:54:36.403; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-11 12:54:36.403; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 17047 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 34963 is already being used by another process (pid: 1348924)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-05-11 12:54:37.569; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-11 12:54:37.572; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 10 post, postlogs and export with basic auth # in 17181 ms
ok 11 package with basic auth # in 11821 ms
ok 12 admin UI static content is excepted from basic auth # in 10229 ms
ok 13 nothing # in 10 ms
ok 14 server does not compress response without Accept-Encoding header # in 174 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 87 ms
ok 16 compressed response can be decompressed and parsed # in 85 ms
ok 17 setting property # in 10800 ms
ok 18 short form of setting property # in 6793 ms
ok 19 connecting to solr via various solr urls and zk hosts # in 9306 ms
ok 20 create for non cloud mode # in 7684 ms
ok 21 create for cloud mode # in 9632 ms
ok 22 multiple connection options are prevented # in 851 ms
ok 23 create collection # in 7466 ms
ok 24 create collection using solrUrl # in 5065 ms
ok 25 create collection using legacy solrUrl # in 5184 ms
ok 26 create collection using Zookeeper # in 5231 ms
ok 27 reject d option with invalid config dir # in 2634 ms
ok 28 accept d option with builtin config # in 5100 ms
ok 29 accept d option with explicit path to config # in 5111 ms
ok 30 accept n option as config name # in 5114 ms
ok 31 allow config reuse when n option specifies same config # in 8836 ms
ok 32 create multisharded collections when s provided # in 5057 ms
ok 33 create replicated collections when rf provided # in 5951 ms
ok 34 can delete collections # in 10213 ms
ok 35 can delete collections with solr-url # in 8082 ms
ok 36 collection delete also deletes zk config # in 6984 ms
ok 37 deletes accompanying zk config with nondefault name # in 7046 ms
ok 38 delete-config option can opt to leave config in zk # in 6973 ms
ok 39 start -e cloud works with --jvm-opts # in 21159 ms
ok 40 start -e cloud works with --no-prompt # in 18306 ms
ok 41 Check export command # in 18420 ms
ok 42 export fails on non cloud mode # in 9212 ms
Tika Server started on port 34968
ok 43 using curl to extract a single pdf file # in 19765 ms
ok 44 using the bin/solr post tool to extract content from pdf # in 20635 ms
ok 45 using the bin/solr post tool to crawl web site # in 22891 ms
Stopping Tika Server container
ok 46 healthcheck on cloud solr # in 10271 ms
ok 47 healthcheck errors on standalone solr # in 11216 ms
ok 48 solr help flag prints help # in 734 ms
ok 49 solr with no flags prints help # in 746 ms
ok 50 start help flag prints help # in 142 ms
ok 51 start h flag prints help # in 167 ms
ok 52 stop help flag prints help # in 146 ms
ok 53 restart help flag prints help # in 157 ms
ok 54 status help flag prints help # in 826 ms
ok 55 healthcheck help flag prints help # in 764 ms
ok 56 create help flag prints help # in 778 ms
ok 57 delete help flag prints help # in 778 ms
ok 58 zk help flag prints help # in 819 ms
ok 59 auth help flag prints help # in 774 ms
ok 60 assert help flag prints help # in 785 ms
ok 61 post help flag prints help # in 743 ms
ok 62 SQL Module # in 15594 ms
ok 63 icu collation in analysis-extras module # in 12392 ms
ok 64 Check lifecycle of sentiment classification # in 23286 ms
ok 65 lifecycle of package # in 8461 ms
ok 66 deploying and undeploying a collection level package # in 13784 ms
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 68 Affinity placement plugin using sysprop # in 15296 ms
ok 69 Random placement plugin using ENV # in 15604 ms
ok 70 Check help command # in 2454 ms
ok 71 basic post with a type specified # in 5404 ms
ok 72 basic post with collection instead of url parameter # in 4041 ms
ok 73 basic post with solr-url and collection # in 4118 ms
ok 74 basic post WITHOUT a type specified # in 12021 ms
ok 75 crawling a directory as a dry-run # in 828 ms
ok 76 crawling a directory # in 4474 ms
ok 77 crawling a web site # in 34330 ms
ok 78 skipcommit and optimize and delete # in 5917 ms
ok 79 args mode # in 9141 ms
not ok 80 stdin mode # in 7900 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 201)
# `assert_output --partial "Created collection 'test_stdin'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'test_stdin'
# output (13 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34963.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34963.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:34963/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:34963 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-11 13:04:40.146; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-11 13:04:40.150; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:34963/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34963.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34963.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:34963/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:34963 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-11 13:04:40.146; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-11 13:04:40.150; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:34963/solr/admin/collections
not ok 81 verbose echo the Solr response # in 1151 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34963.
#
# ERROR: Server refused connection at: http://localhost:34963/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34963.
#
# ERROR: Server refused connection at: http://localhost:34963/solr/admin/info/system
ok 82 post solr log into solr via cli # in 9556 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 16 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-11T13:05:12.477177726Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-11T13:05:12.477210060Z It should be set to 65000 to avoid operational disruption.
2026-05-11T13:05:12.477213183Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-11T13:05:12.477705882Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-11T13:05:12.477717982Z
2026-05-11T13:05:12.487860173Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-11T13:05:12.495172693Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-11T13:05:12.545381927Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-11T13:05:12.584961703Z WARNING: A command line option has enabled the Security Manager
2026-05-11T13:05:12.584985068Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-11T13:05:13.659646358Z 2026-05-11 13:05:13.656 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-11T13:05:13.937106534Z 2026-05-11 13:05:13.936 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-11T13:05:13.962197227Z 2026-05-11 13:05:13.961 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-11T13:05:13.969258345Z 2026-05-11 13:05:13.968 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-11T13:05:13.969319381Z 2026-05-11 13:05:13.968 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-11T13:05:13.969467342Z 2026-05-11 13:05:13.968 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-11T13:05:13.969591609Z 2026-05-11 13:05:13.968 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-11T13:05:13.968942549Z
2026-05-11T13:05:13.972152555Z 2026-05-11 13:05:13.971 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-11T13:05:13.994426844Z 2026-05-11 13:05:13.993 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-11T13:05:14.008718789Z 2026-05-11 13:05:14.008 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-11T13:05:14.008738926Z 2026-05-11 13:05:14.008 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-11T13:05:14.045007151Z 2026-05-11 13:05:14.044 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-11T13:05:14.759808400Z 2026-05-11 13:05:14.759 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-11T13:05:15.028135342Z 2026-05-11 13:05:15.027 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-11T13:05:15.033621160Z 2026-05-11 13:05:15.033 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-11T13:05:15.033749339Z 2026-05-11 13:05:15.033 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-11T13:05:15.036244304Z 2026-05-11 13:05:15.035 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-11T13:05:15.036381277Z 2026-05-11 13:05:15.035 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-11T13:05:15.168121966Z 2026-05-11 13:05:15.167 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-11T13:05:15.538585606Z 2026-05-11 13:05:15.538 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-11T13:05:15.552225977Z 2026-05-11 13:05:15.551 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-11T13:05:15.609208321Z 2026-05-11 13:05:15.608 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-11T13:05:15.634585348Z 2026-05-11 13:05:15.634 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-11T13:05:15.634698289Z 2026-05-11 13:05:15.634 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-11T13:05:15.785402924Z 2026-05-11 13:05:15.783 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-11T13:05:15.811267096Z 2026-05-11 13:05:15.810 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-11T13:05:15.826425436Z 2026-05-11 13:05:15.825 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-11T13:05:15.833678564Z 2026-05-11 13:05:15.833 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72115856288382976-solr-node1:8983_solr-n_0000000000) starting
2026-05-11T13:05:15.914129512Z 2026-05-11 13:05:15.913 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-11T13:05:15.914846743Z 2026-05-11 13:05:15.914 INFO (OverseerStateUpdate-72115856288382976-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-11T13:05:15.926898202Z 2026-05-11 13:05:15.926 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-11T13:05:15.981805780Z 2026-05-11 13:05:15.981 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-11T13:05:16.432457006Z 2026-05-11 13:05:16.431 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-11T13:05:16.614236358Z 2026-05-11 13:05:16.613 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-11T13:05:16.882547163Z 2026-05-11 13:05:16.882 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-11T13:05:17.018097095Z 2026-05-11 13:05:17.017 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-11T13:05:17.023871343Z 2026-05-11 13:05:17.023 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_11.request.log
2026-05-11T13:05:17.064981461Z 2026-05-11 13:05:17.063 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-11T13:05:17.065437007Z 2026-05-11 13:05:17.065 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4571ms
2026-05-11T13:05:17.236153875Z 2026-05-11 13:05:17.235 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-05-11T13:05:17.299743974Z 2026-05-11 13:05:17.299 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-05-11T13:05:20.778040687Z 2026-05-11 13:05:20.777 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-11T13:05:20.867359775Z 2026-05-11 13:05:20.866 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-11T13:05:24.777148326Z 2026-05-11 13:05:24.776 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-05-11T13:05:24.808833149Z 2026-05-11 13:05:24.808 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-05-11T13:05:24.837891981Z 2026-05-11 13:05:24.837 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-11T13:05:25.157841360Z 2026-05-11 13:05:25.157 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-11T13:05:25.177930001Z 2026-05-11 13:05:25.177 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-11T13:05:25.184939755Z 2026-05-11 13:05:25.184 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:25.226846055Z 2026-05-11 13:05:25.226 INFO (OverseerStateUpdate-72115856288382976-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-11T13:05:25.226867296Z "core":"test-collection_shard1_replica_n1",
2026-05-11T13:05:25.226870692Z "node_name":"solr-node2:8984_solr",
2026-05-11T13:05:25.226873410Z "base_url":"http://solr-node2:8984/solr",
2026-05-11T13:05:25.226876026Z "collection":"test-collection",
2026-05-11T13:05:25.226878731Z "shard":"shard1",
2026-05-11T13:05:25.226881238Z "state":"down",
2026-05-11T13:05:25.226883691Z "type":"NRT",
2026-05-11T13:05:25.226886130Z "operation":"ADDREPLICA",
2026-05-11T13:05:25.226888725Z "waitForFinalState":"false"}
2026-05-11T13:05:25.234113649Z 2026-05-11 13:05:25.233 INFO (OverseerStateUpdate-72115856288382976-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-11T13:05:25.234131767Z "core":"test-collection_shard2_replica_n2",
2026-05-11T13:05:25.234134984Z "node_name":"solr-node3:8985_solr",
2026-05-11T13:05:25.234138313Z "base_url":"http://solr-node3:8985/solr",
2026-05-11T13:05:25.234141026Z "collection":"test-collection",
2026-05-11T13:05:25.234143529Z "shard":"shard2",
2026-05-11T13:05:25.234145955Z "state":"down",
2026-05-11T13:05:25.234149888Z "type":"NRT",
2026-05-11T13:05:25.234152415Z "operation":"ADDREPLICA",
2026-05-11T13:05:25.234154924Z "waitForFinalState":"false"}
2026-05-11T13:05:25.250499054Z 2026-05-11 13:05:25.250 INFO (OverseerStateUpdate-72115856288382976-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-11T13:05:25.250523804Z "core":"test-collection_shard3_replica_n4",
2026-05-11T13:05:25.250527422Z "node_name":"solr-node1:8983_solr",
2026-05-11T13:05:25.250540272Z "base_url":"http://solr-node1:8983/solr",
2026-05-11T13:05:25.250542479Z "collection":"test-collection",
2026-05-11T13:05:25.250544535Z "shard":"shard3",
2026-05-11T13:05:25.250546470Z "state":"down",
2026-05-11T13:05:25.250548432Z "type":"NRT",
2026-05-11T13:05:25.250550303Z "operation":"ADDREPLICA",
2026-05-11T13:05:25.250552302Z "waitForFinalState":"false"}
2026-05-11T13:05:25.361391830Z 2026-05-11 13:05:25.360 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:25.481367587Z 2026-05-11 13:05:25.478 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-11T13:05:25.530063280Z 2026-05-11 13:05:25.529 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-11T13:05:25.644401970Z 2026-05-11 13:05:25.643 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-11T13:05:26.072070075Z 2026-05-11 13:05:26.070 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-11T13:05:26.072095265Z 2026-05-11 13:05:26.071 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-11T13:05:26.099343234Z 2026-05-11 13:05:26.098 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n4], dataDir=[/var/solr/data/test-collection_shard3_replica_n4/data/]
2026-05-11T13:05:27.800930011Z 2026-05-11 13:05:27.799 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-11T13:05:27.870805590Z 2026-05-11 13:05:27.870 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-11T13:05:27.880693058Z 2026-05-11 13:05:27.880 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-11T13:05:27.882607275Z 2026-05-11 13:05:27.881 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-11T13:05:27.882764123Z 2026-05-11 13:05:27.882 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-11T13:05:27.969062791Z 2026-05-11 13:05:27.968 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-11T13:05:27.972501892Z 2026-05-11 13:05:27.972 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-11T13:05:27.974319114Z 2026-05-11 13:05:27.974 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-11T13:05:27.976468376Z 2026-05-11 13:05:27.976 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-11T13:05:27.991335047Z 2026-05-11 13:05:27.990 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-11T13:05:27.998943033Z 2026-05-11 13:05:27.998 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-11T13:05:28.004970760Z 2026-05-11 13:05:28.004 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-11T13:05:28.004986695Z 2026-05-11 13:05:28.004 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-11T13:05:28.015631856Z 2026-05-11 13:05:28.015 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-11T13:05:28.022016896Z 2026-05-11 13:05:28.021 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-11T13:05:28.023720306Z 2026-05-11 13:05:28.023 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-11T13:05:28.044028282Z 2026-05-11 13:05:28.040 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-11T13:05:28.045933127Z 2026-05-11 13:05:28.040 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-11T13:05:28.047916986Z 2026-05-11 13:05:28.040 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n4/
2026-05-11T13:05:28.049929928Z 2026-05-11 13:05:28.042 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-11T13:05:28.052962565Z 2026-05-11 13:05:28.042 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ has no replicas
2026-05-11T13:05:28.055014369Z 2026-05-11 13:05:28.044 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72115856288382976-core_node6-n_0000000000
2026-05-11T13:05:28.057925118Z 2026-05-11 13:05:28.056 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ shard3
2026-05-11T13:05:28.162239563Z 2026-05-11 13:05:28.161 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:28.168883589Z 2026-05-11 13:05:28.168 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-11T13:05:28.179146527Z 2026-05-11 13:05:28.178 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2702
2026-05-11T13:05:28.277583313Z 2026-05-11 13:05:28.277 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1381191,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-11T13:05:12.234401758Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-11T13:05:17.631062074Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-11T13:05:17.631087847Z It should be set to 65000 to avoid operational disruption.
2026-05-11T13:05:17.631091084Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-11T13:05:17.631399409Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-11T13:05:17.631408371Z
2026-05-11T13:05:17.641200098Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-11T13:05:17.648995611Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-11T13:05:17.694329930Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-11T13:05:17.728923157Z WARNING: A command line option has enabled the Security Manager
2026-05-11T13:05:17.728940741Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-11T13:05:18.973217036Z 2026-05-11 13:05:18.970 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-11T13:05:19.247096759Z 2026-05-11 13:05:19.246 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-11T13:05:19.273296745Z 2026-05-11 13:05:19.272 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-11T13:05:19.280228059Z 2026-05-11 13:05:19.279 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-11T13:05:19.280353235Z 2026-05-11 13:05:19.279 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-11T13:05:19.280467365Z 2026-05-11 13:05:19.280 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-11T13:05:19.280584766Z 2026-05-11 13:05:19.280 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-11T13:05:19.280080837Z
2026-05-11T13:05:19.283194126Z 2026-05-11 13:05:19.282 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-11T13:05:19.306164493Z 2026-05-11 13:05:19.305 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-11T13:05:19.330363434Z 2026-05-11 13:05:19.329 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-11T13:05:19.489065244Z 2026-05-11 13:05:19.488 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-11T13:05:19.508863309Z 2026-05-11 13:05:19.508 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-11T13:05:19.509055882Z 2026-05-11 13:05:19.508 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-11T13:05:19.509156727Z 2026-05-11 13:05:19.508 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-11T13:05:19.619182071Z 2026-05-11 13:05:19.618 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-11T13:05:19.619359582Z 2026-05-11 13:05:19.619 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-11T13:05:19.619501360Z 2026-05-11 13:05:19.619 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-11T13:05:19.666480094Z 2026-05-11 13:05:19.665 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-11T13:05:20.525408462Z 2026-05-11 13:05:20.524 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-11T13:05:20.669501828Z 2026-05-11 13:05:20.669 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-11T13:05:20.671047370Z 2026-05-11 13:05:20.669 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-11T13:05:20.722872670Z 2026-05-11 13:05:20.720 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-11T13:05:20.747005142Z 2026-05-11 13:05:20.746 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-11T13:05:20.756865281Z 2026-05-11 13:05:20.756 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-11T13:05:20.757024626Z 2026-05-11 13:05:20.756 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-11T13:05:20.795699477Z 2026-05-11 13:05:20.795 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-11T13:05:20.806493381Z 2026-05-11 13:05:20.805 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-11T13:05:20.829261734Z 2026-05-11 13:05:20.828 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-11T13:05:20.845027656Z 2026-05-11 13:05:20.844 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-11T13:05:20.858369336Z 2026-05-11 13:05:20.857 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-11T13:05:20.868833681Z 2026-05-11 13:05:20.868 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-11T13:05:20.973161685Z 2026-05-11 13:05:20.972 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-11T13:05:21.264318344Z 2026-05-11 13:05:21.263 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-11T13:05:21.439845461Z 2026-05-11 13:05:21.439 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-11T13:05:21.711584268Z 2026-05-11 13:05:21.710 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-11T13:05:21.877249120Z 2026-05-11 13:05:21.876 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-11T13:05:21.882464018Z 2026-05-11 13:05:21.882 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_11.request.log
2026-05-11T13:05:21.887048515Z 2026-05-11 13:05:21.886 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-11T13:05:21.888689325Z 2026-05-11 13:05:21.888 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4241ms
2026-05-11T13:05:22.055020162Z 2026-05-11 13:05:22.054 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=45
2026-05-11T13:05:22.115689215Z 2026-05-11 13:05:22.114 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-11T13:05:25.479848470Z 2026-05-11 13:05:25.479 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard1_replica_n1&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-11T13:05:25.529756965Z 2026-05-11 13:05:25.529 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-11T13:05:25.586348533Z 2026-05-11 13:05:25.585 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-11T13:05:25.923519084Z 2026-05-11 13:05:25.923 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-11T13:05:25.923557195Z 2026-05-11 13:05:25.923 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-11T13:05:25.942459002Z 2026-05-11 13:05:25.942 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n1], dataDir=[/var/solr/data/test-collection_shard1_replica_n1/data/]
2026-05-11T13:05:26.380674676Z 2026-05-11 13:05:26.380 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-11T13:05:26.400305356Z 2026-05-11 13:05:26.399 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-11T13:05:26.403671555Z 2026-05-11 13:05:26.403 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-11T13:05:26.404886132Z 2026-05-11 13:05:26.404 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-11T13:05:26.404902370Z 2026-05-11 13:05:26.404 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-11T13:05:26.445358429Z 2026-05-11 13:05:26.444 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-11T13:05:26.448390066Z 2026-05-11 13:05:26.448 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-11T13:05:26.450499359Z 2026-05-11 13:05:26.450 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-11T13:05:26.452321438Z 2026-05-11 13:05:26.452 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-11T13:05:26.463773268Z 2026-05-11 13:05:26.463 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-11T13:05:26.471037255Z 2026-05-11 13:05:26.470 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-11T13:05:26.476982378Z 2026-05-11 13:05:26.476 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node3 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-11T13:05:26.477027456Z 2026-05-11 13:05:26.476 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node3 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-11T13:05:26.484525897Z 2026-05-11 13:05:26.484 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node3 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-11T13:05:26.503258792Z 2026-05-11 13:05:26.502 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-11T13:05:26.504659334Z 2026-05-11 13:05:26.504 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-11T13:05:26.540515161Z 2026-05-11 13:05:26.540 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-11T13:05:26.540618512Z 2026-05-11 13:05:26.540 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-11T13:05:26.540784470Z 2026-05-11 13:05:26.540 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n1/
2026-05-11T13:05:26.542206949Z 2026-05-11 13:05:26.541 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-11T13:05:26.542836343Z 2026-05-11 13:05:26.542 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n1/ has no replicas
2026-05-11T13:05:26.543937662Z 2026-05-11 13:05:26.543 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72115856288382980-core_node3-n_0000000000
2026-05-11T13:05:26.856492787Z 2026-05-11 13:05:26.856 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n1/ shard1
2026-05-11T13:05:26.983686886Z 2026-05-11 13:05:26.983 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 207] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:27.016293229Z 2026-05-11 13:05:27.015 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node3 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-11T13:05:27.043103979Z 2026-05-11 13:05:27.042 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1565
2026-05-11T13:05:27.191613230Z 2026-05-11 13:05:27.190 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 221] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:27.304535734Z 2026-05-11 13:05:27.304 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:28.162462516Z 2026-05-11 13:05:28.162 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:28.277630440Z 2026-05-11 13:05:28.277 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1381585,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-11T13:05:17.402270613Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-11T13:05:17.861396290Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-11T13:05:17.861452095Z It should be set to 65000 to avoid operational disruption.
2026-05-11T13:05:17.861456841Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-11T13:05:17.861696718Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-11T13:05:17.861708970Z
2026-05-11T13:05:17.872370371Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-11T13:05:17.880746819Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-11T13:05:17.927325295Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-11T13:05:17.962738875Z WARNING: A command line option has enabled the Security Manager
2026-05-11T13:05:17.962771476Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-11T13:05:19.122314003Z 2026-05-11 13:05:19.118 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-11T13:05:19.388028222Z 2026-05-11 13:05:19.387 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-11T13:05:19.413937581Z 2026-05-11 13:05:19.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-11T13:05:19.420968183Z 2026-05-11 13:05:19.420 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-11T13:05:19.421289863Z 2026-05-11 13:05:19.420 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-11T13:05:19.421301375Z 2026-05-11 13:05:19.420 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-11T13:05:19.421304610Z 2026-05-11 13:05:19.420 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-11T13:05:19.420684980Z
2026-05-11T13:05:19.423921816Z 2026-05-11 13:05:19.423 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-11T13:05:19.445323698Z 2026-05-11 13:05:19.444 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-11T13:05:19.468489151Z 2026-05-11 13:05:19.467 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-11T13:05:19.614258318Z 2026-05-11 13:05:19.613 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-11T13:05:19.633059986Z 2026-05-11 13:05:19.632 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-11T13:05:19.633148418Z 2026-05-11 13:05:19.632 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-11T13:05:19.633327467Z 2026-05-11 13:05:19.632 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-11T13:05:19.743423943Z 2026-05-11 13:05:19.742 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-11T13:05:19.743534545Z 2026-05-11 13:05:19.743 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-11T13:05:19.743615608Z 2026-05-11 13:05:19.743 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-11T13:05:19.778850851Z 2026-05-11 13:05:19.778 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-11T13:05:20.529021092Z 2026-05-11 13:05:20.528 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-11T13:05:20.668803813Z 2026-05-11 13:05:20.668 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-11T13:05:20.670740956Z 2026-05-11 13:05:20.668 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-11T13:05:20.678497108Z 2026-05-11 13:05:20.678 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-11T13:05:20.683530290Z 2026-05-11 13:05:20.683 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-11T13:05:20.687247680Z 2026-05-11 13:05:20.686 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-11T13:05:20.687259092Z 2026-05-11 13:05:20.686 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-11T13:05:20.718653306Z 2026-05-11 13:05:20.718 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-11T13:05:20.727035941Z 2026-05-11 13:05:20.726 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-11T13:05:20.746637804Z 2026-05-11 13:05:20.746 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-11T13:05:20.757997787Z 2026-05-11 13:05:20.757 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-11T13:05:20.769688271Z 2026-05-11 13:05:20.769 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-11T13:05:20.779315799Z 2026-05-11 13:05:20.778 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-11T13:05:20.824548319Z 2026-05-11 13:05:20.823 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-11T13:05:20.867683631Z 2026-05-11 13:05:20.867 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-11T13:05:21.096513787Z 2026-05-11 13:05:21.095 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-11T13:05:21.261952821Z 2026-05-11 13:05:21.261 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-11T13:05:21.514010653Z 2026-05-11 13:05:21.513 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-11T13:05:21.647903086Z 2026-05-11 13:05:21.646 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-11T13:05:21.656547871Z 2026-05-11 13:05:21.656 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_11.request.log
2026-05-11T13:05:21.660035455Z 2026-05-11 13:05:21.659 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-11T13:05:21.660805274Z 2026-05-11 13:05:21.660 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3781ms
2026-05-11T13:05:23.625930933Z 2026-05-11 13:05:23.617 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=69
2026-05-11T13:05:23.700956421Z 2026-05-11 13:05:23.700 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=22
2026-05-11T13:05:25.103604785Z 2026-05-11 13:05:25.103 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=8
2026-05-11T13:05:25.480965474Z 2026-05-11 13:05:25.480 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-11T13:05:25.554039806Z 2026-05-11 13:05:25.553 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-11T13:05:25.626436510Z 2026-05-11 13:05:25.625 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-11T13:05:26.075809563Z 2026-05-11 13:05:26.075 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-11T13:05:26.075922019Z 2026-05-11 13:05:26.075 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-11T13:05:26.095426121Z 2026-05-11 13:05:26.094 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-11T13:05:26.533150532Z 2026-05-11 13:05:26.532 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-11T13:05:26.553150851Z 2026-05-11 13:05:26.552 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-11T13:05:26.823458350Z 2026-05-11 13:05:26.822 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-11T13:05:26.824711712Z 2026-05-11 13:05:26.824 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-11T13:05:26.824725308Z 2026-05-11 13:05:26.824 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-11T13:05:26.907269807Z 2026-05-11 13:05:26.906 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-11T13:05:26.912166705Z 2026-05-11 13:05:26.910 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-11T13:05:26.914828749Z 2026-05-11 13:05:26.914 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-11T13:05:26.917067291Z 2026-05-11 13:05:26.916 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-11T13:05:26.931682882Z 2026-05-11 13:05:26.931 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-11T13:05:26.939115913Z 2026-05-11 13:05:26.938 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-11T13:05:26.945325879Z 2026-05-11 13:05:26.944 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-11T13:05:26.945410456Z 2026-05-11 13:05:26.945 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-11T13:05:26.957460096Z 2026-05-11 13:05:26.957 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-11T13:05:27.024925665Z 2026-05-11 13:05:27.024 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-11T13:05:27.026772670Z 2026-05-11 13:05:27.026 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-11T13:05:27.064633811Z 2026-05-11 13:05:27.064 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-11T13:05:27.064653534Z 2026-05-11 13:05:27.064 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-11T13:05:27.064656711Z 2026-05-11 13:05:27.064 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-11T13:05:27.066392298Z 2026-05-11 13:05:27.066 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-11T13:05:27.067104123Z 2026-05-11 13:05:27.066 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-11T13:05:27.068507957Z 2026-05-11 13:05:27.068 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72115856288382979-core_node5-n_0000000000
2026-05-11T13:05:27.086029588Z 2026-05-11 13:05:27.085 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-11T13:05:27.191825940Z 2026-05-11 13:05:27.191 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 221] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:27.195476590Z 2026-05-11 13:05:27.195 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-11T13:05:27.204127642Z 2026-05-11 13:05:27.203 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1725
2026-05-11T13:05:27.304481801Z 2026-05-11 13:05:27.304 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:28.162603872Z 2026-05-11 13:05:28.162 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:28.235725589Z 2026-05-11 13:05:28.235 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-11T13:05:28.277678822Z 2026-05-11 13:05:28.277 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-11T13:05:28.280240760Z 2026-05-11 13:05:28.279 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3163
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1381769,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-11T13:05:17.639603563Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 27254 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# 87f3bb8db2f4: Already exists
# 7c0b62ebdf15: Already exists
# 5b95479e5f9b: Already exists
# c913227527c9: Already exists
# 5eae1a236a54: Pulling fs layer
# 197c7ec22b6b: Pulling fs layer
# 8eed92533426: Pulling fs layer
# d731976aba0c: Pulling fs layer
# 2a380a22a70b: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# d731976aba0c: Waiting
# 2a380a22a70b: Waiting
# 4f4fb700ef54: Waiting
# 197c7ec22b6b: Verifying Checksum
# 197c7ec22b6b: Download complete
# 8eed92533426: Download complete
# d731976aba0c: Verifying Checksum
# d731976aba0c: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 2a380a22a70b: Verifying Checksum
# 2a380a22a70b: Download complete
# 5eae1a236a54: Verifying Checksum
# 5eae1a236a54: Download complete
# 5eae1a236a54: Pull complete
# 197c7ec22b6b: Pull complete
# 8eed92533426: Pull complete
# d731976aba0c: Pull complete
# 2a380a22a70b: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:75dc25ea19e534f02fd5f47fb42a4c832d9b24f6de87c6adc1b75bf0cd330e1c
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 6e2e944814f24e912271ff5a54aa312d88806ee45e8ae6331db6920655b6b814
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# c6f754cb55c18601f8bcad0acaba533e48d10eb8a14c01fecda2006901a3bfbf
# Starting solr-node2 connected to first node's ZooKeeper
# 00713af08906c077204eaa56153ebd95e9c8ea8d46d3c44229588f92294435c6
# Started solr-node3 connected to first node's ZooKeeper
# 82bbe8bca7508aea4ec8c0440f08f9b2295f6dd9bfc014089415506688deb2cf
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-11 13:05:24.868; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-11 13:05:24.900; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 15616 ms
ok 85 snapshot lifecycle # in 4831 ms
ok 86 snapshot list # in 3382 ms
ok 87 snapshot describe # in 3479 ms
ok 88 start solr with ssl # in 16780 ms
ok 89 use different hostname when not checking peer-name # in 26059 ms
ok 90 start solr with ssl and auth # in 15175 ms
ok 91 start solr with client truststore and security manager # in 16846 ms
ok 92 start solr with mTLS needed # in 46470 ms
ok 93 start solr with mTLS wanted # in 34519 ms
ok 94 test keystore reload # in 59292 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15963 ms
ok 96 stop command for single port # in 18454 ms
ok 97 check stop command doesn't hang # in 227 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8072 ms
ok 99 deprecated system properties converted to modern properties # in 195170 ms
ok 100 start with custom jetty options # in 8118 ms
ok 101 bootstrapping a configset # in 9209 ms
ok 102 status detects locally running solr # in 9439 ms
ok 103 status with --solr-url from user # in 8011 ms
ok 104 status with --port from user # in 7947 ms
ok 105 multiple connection options are prevented # in 859 ms
ok 106 status with invalid --solr-url from user # in 1242 ms
ok 107 status with --short format # in 195905 ms
ok 108 searching solr via locally executed streaming expression # in 1708 ms
ok 109 searching solr via remotely executed streaming expression # in 1262 ms
ok 110 variable interpolation # in 1698 ms
ok 111 searching solr without credentials fails with error # in 1114 ms
ok 112 --version returns Solr version # in 740 ms
ok 113 version as direct tool call still runs # in 799 ms
ok 114 short help # in 874 ms
ok 115 short help is inferred # in 794 ms
ok 116 long help # in 779 ms
ok 117 running subcommands with zk is prevented # in 769 ms
ok 118 listing out files # in 2146 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7797 ms
ok 120 copying files around # in 10981 ms
ok 121 upconfig # in 2525 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1209 ms
ok 123 downconfig # in 1199 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1433 ms
ok 125 env var ZK_HOST is honored # in 3199 ms
ok 126 Cleanup # in 1896 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 42s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/qaxoiy3j4yspq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 205 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/205/
No tests ran.
Build Log:
[...truncated 417 lines...]
# ERROR: Server refused connection at: http://localhost:39595/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:39595/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# POSTing stdin to http://localhost:39595/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# COMMITting Solr index changes to http://localhost:39595/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:39595/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:39595/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 81 verbose echo the Solr response # in 1089 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39595.
#
# ERROR: Server refused connection at: http://localhost:39595/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39595.
#
# ERROR: Server refused connection at: http://localhost:39595/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8791 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 13 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 26 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-10T17:31:38.982871202Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-10T17:31:38.982903409Z It should be set to 65000 to avoid operational disruption.
2026-05-10T17:31:38.982906394Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-10T17:31:38.983151920Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-10T17:31:38.983157210Z
2026-05-10T17:31:38.996250406Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-10T17:31:39.006115375Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-10T17:31:39.051251864Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-10T17:31:39.132672721Z WARNING: A command line option has enabled the Security Manager
2026-05-10T17:31:39.132692402Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-10T17:31:41.497846717Z 2026-05-10 17:31:41.494 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-10T17:31:42.055593182Z 2026-05-10 17:31:42.055 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-10T17:31:42.206827662Z 2026-05-10 17:31:42.206 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-10T17:31:42.218976547Z 2026-05-10 17:31:42.218 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-10T17:31:42.219036891Z 2026-05-10 17:31:42.218 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-10T17:31:42.219259673Z 2026-05-10 17:31:42.218 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-10T17:31:42.219444599Z 2026-05-10 17:31:42.218 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-10T17:31:42.218714729Z
2026-05-10T17:31:42.224368883Z 2026-05-10 17:31:42.223 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-10T17:31:42.270549969Z 2026-05-10 17:31:42.270 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-10T17:31:42.301573518Z 2026-05-10 17:31:42.300 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-10T17:31:42.301591628Z 2026-05-10 17:31:42.301 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-10T17:31:42.500707685Z 2026-05-10 17:31:42.500 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-10T17:31:44.247624767Z 2026-05-10 17:31:44.247 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-10T17:31:44.859320218Z 2026-05-10 17:31:44.858 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-10T17:31:44.871743648Z 2026-05-10 17:31:44.871 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-10T17:31:44.871831121Z 2026-05-10 17:31:44.871 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-10T17:31:44.879975048Z 2026-05-10 17:31:44.879 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-10T17:31:44.880052760Z 2026-05-10 17:31:44.879 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-10T17:31:44.950452007Z 2026-05-10 17:31:44.949 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-10T17:31:45.382694109Z 2026-05-10 17:31:45.382 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-10T17:31:45.428942810Z 2026-05-10 17:31:45.428 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-10T17:31:45.507562846Z 2026-05-10 17:31:45.507 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-10T17:31:45.532575622Z 2026-05-10 17:31:45.532 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-10T17:31:45.533301557Z 2026-05-10 17:31:45.532 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-10T17:31:45.650777052Z 2026-05-10 17:31:45.650 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-10T17:31:45.677166418Z 2026-05-10 17:31:45.676 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-10T17:31:45.687459414Z 2026-05-10 17:31:45.687 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-10T17:31:45.693919134Z 2026-05-10 17:31:45.693 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72111156159447040-solr-node1:8983_solr-n_0000000000) starting
2026-05-10T17:31:45.784527698Z 2026-05-10 17:31:45.784 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-10T17:31:45.786001350Z 2026-05-10 17:31:45.785 INFO (OverseerStateUpdate-72111156159447040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-10T17:31:45.797987873Z 2026-05-10 17:31:45.797 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-10T17:31:45.925945008Z 2026-05-10 17:31:45.925 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-10T17:31:46.624562190Z 2026-05-10 17:31:46.624 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-10T17:31:46.936613196Z 2026-05-10 17:31:46.936 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-10T17:31:47.318608355Z 2026-05-10 17:31:47.318 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-10T17:31:47.808200564Z 2026-05-10 17:31:47.807 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-10T17:31:47.841110219Z 2026-05-10 17:31:47.840 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_10.request.log
2026-05-10T17:31:47.854569790Z 2026-05-10 17:31:47.853 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-10T17:31:47.857162493Z 2026-05-10 17:31:47.856 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @8852ms
2026-05-10T17:31:48.579906128Z 2026-05-10 17:31:48.577 INFO (qtp985324122-28-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=95
2026-05-10T17:31:48.696938808Z 2026-05-10 17:31:48.696 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=20
2026-05-10T17:31:53.275156088Z 2026-05-10 17:31:53.274 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-10T17:31:53.425117702Z 2026-05-10 17:31:53.424 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-10T17:31:58.768761209Z 2026-05-10 17:31:58.767 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=10
2026-05-10T17:31:58.834112506Z 2026-05-10 17:31:58.833 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=14
2026-05-10T17:31:58.853504944Z 2026-05-10 17:31:58.853 INFO (qtp985324122-28-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-10T17:31:59.196029120Z 2026-05-10 17:31:59.195 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-10T17:31:59.221764200Z 2026-05-10 17:31:59.221 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-10T17:31:59.272063756Z 2026-05-10 17:31:59.268 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:31:59.337373377Z 2026-05-10 17:31:59.336 INFO (OverseerStateUpdate-72111156159447040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-10T17:31:59.337393880Z "core":"test-collection_shard3_replica_n1",
2026-05-10T17:31:59.337396739Z "node_name":"solr-node1:8983_solr",
2026-05-10T17:31:59.337402711Z "base_url":"http://solr-node1:8983/solr",
2026-05-10T17:31:59.337404976Z "collection":"test-collection",
2026-05-10T17:31:59.337407074Z "shard":"shard3",
2026-05-10T17:31:59.337409049Z "state":"down",
2026-05-10T17:31:59.337411494Z "type":"NRT",
2026-05-10T17:31:59.337413933Z "operation":"ADDREPLICA",
2026-05-10T17:31:59.337415949Z "waitForFinalState":"false"}
2026-05-10T17:31:59.402229151Z 2026-05-10 17:31:59.401 INFO (OverseerStateUpdate-72111156159447040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-10T17:31:59.402251169Z "core":"test-collection_shard2_replica_n2",
2026-05-10T17:31:59.402254198Z "node_name":"solr-node3:8985_solr",
2026-05-10T17:31:59.402256622Z "base_url":"http://solr-node3:8985/solr",
2026-05-10T17:31:59.402258775Z "collection":"test-collection",
2026-05-10T17:31:59.402260761Z "shard":"shard2",
2026-05-10T17:31:59.402262702Z "state":"down",
2026-05-10T17:31:59.402264753Z "type":"NRT",
2026-05-10T17:31:59.402266716Z "operation":"ADDREPLICA",
2026-05-10T17:31:59.402268634Z "waitForFinalState":"false"}
2026-05-10T17:31:59.405882095Z 2026-05-10 17:31:59.405 INFO (OverseerStateUpdate-72111156159447040-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-10T17:31:59.405901232Z "core":"test-collection_shard1_replica_n4",
2026-05-10T17:31:59.405904146Z "node_name":"solr-node2:8984_solr",
2026-05-10T17:31:59.405918961Z "base_url":"http://solr-node2:8984/solr",
2026-05-10T17:31:59.405921161Z "collection":"test-collection",
2026-05-10T17:31:59.405923137Z "shard":"shard1",
2026-05-10T17:31:59.405925136Z "state":"down",
2026-05-10T17:31:59.405927002Z "type":"NRT",
2026-05-10T17:31:59.405928896Z "operation":"ADDREPLICA",
2026-05-10T17:31:59.405930933Z "waitForFinalState":"false"}
2026-05-10T17:31:59.513780098Z 2026-05-10 17:31:59.513 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 187] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:31:59.602992290Z 2026-05-10 17:31:59.602 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-10T17:31:59.662069533Z 2026-05-10 17:31:59.661 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-10T17:31:59.721105924Z 2026-05-10 17:31:59.720 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-10T17:32:00.317434582Z 2026-05-10 17:32:00.316 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-10T17:32:00.317618369Z 2026-05-10 17:32:00.317 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-10T17:32:00.383658067Z 2026-05-10 17:32:00.383 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-10T17:32:02.178797318Z 2026-05-10 17:32:02.177 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-10T17:32:02.242734944Z 2026-05-10 17:32:02.242 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-10T17:32:02.250550798Z 2026-05-10 17:32:02.249 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-10T17:32:02.251619104Z 2026-05-10 17:32:02.250 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-10T17:32:02.251768507Z 2026-05-10 17:32:02.251 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-10T17:32:02.319966951Z 2026-05-10 17:32:02.319 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-10T17:32:02.323322080Z 2026-05-10 17:32:02.323 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-10T17:32:02.326497162Z 2026-05-10 17:32:02.326 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-10T17:32:02.331132603Z 2026-05-10 17:32:02.330 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-10T17:32:02.352181782Z 2026-05-10 17:32:02.351 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-10T17:32:02.368690734Z 2026-05-10 17:32:02.368 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-10T17:32:02.383545077Z 2026-05-10 17:32:02.382 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-10T17:32:02.383601262Z 2026-05-10 17:32:02.382 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-10T17:32:02.395018821Z 2026-05-10 17:32:02.394 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-10T17:32:02.407265376Z 2026-05-10 17:32:02.406 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-10T17:32:02.413393768Z 2026-05-10 17:32:02.413 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-10T17:32:02.432986285Z 2026-05-10 17:32:02.432 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-10T17:32:02.433047090Z 2026-05-10 17:32:02.432 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-10T17:32:02.433052020Z 2026-05-10 17:32:02.432 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-10T17:32:02.434908489Z 2026-05-10 17:32:02.434 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-10T17:32:02.435557361Z 2026-05-10 17:32:02.435 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-10T17:32:02.437464167Z 2026-05-10 17:32:02.437 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72111156159447040-core_node3-n_0000000000
2026-05-10T17:32:02.444333541Z 2026-05-10 17:32:02.443 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-10T17:32:02.552910023Z 2026-05-10 17:32:02.552 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:02.556637207Z 2026-05-10 17:32:02.556 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-10T17:32:02.564220905Z 2026-05-10 17:32:02.563 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2962
2026-05-10T17:32:02.663418595Z 2026-05-10 17:32:02.662 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3182653,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-10T17:31:36.563884185Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-10T17:31:49.375361523Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-10T17:31:49.375392185Z It should be set to 65000 to avoid operational disruption.
2026-05-10T17:31:49.375395195Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-10T17:31:49.375631206Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-10T17:31:49.375634686Z
2026-05-10T17:31:49.384664490Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-10T17:31:49.393267747Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-10T17:31:49.471169625Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-10T17:31:49.506767452Z WARNING: A command line option has enabled the Security Manager
2026-05-10T17:31:49.506790762Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-10T17:31:50.842075829Z 2026-05-10 17:31:50.838 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-10T17:31:51.232801041Z 2026-05-10 17:31:51.232 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-10T17:31:51.266484031Z 2026-05-10 17:31:51.266 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-10T17:31:51.286317992Z 2026-05-10 17:31:51.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-10T17:31:51.286424368Z 2026-05-10 17:31:51.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-10T17:31:51.286520206Z 2026-05-10 17:31:51.285 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-10T17:31:51.286642560Z 2026-05-10 17:31:51.286 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-10T17:31:51.286106250Z
2026-05-10T17:31:51.289829643Z 2026-05-10 17:31:51.289 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-10T17:31:51.315427599Z 2026-05-10 17:31:51.314 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-10T17:31:51.359279881Z 2026-05-10 17:31:51.358 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-10T17:31:51.609420009Z 2026-05-10 17:31:51.608 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-10T17:31:51.632277771Z 2026-05-10 17:31:51.631 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-10T17:31:51.632403624Z 2026-05-10 17:31:51.631 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-10T17:31:51.632642628Z 2026-05-10 17:31:51.631 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-10T17:31:51.744314618Z 2026-05-10 17:31:51.743 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-10T17:31:51.744888648Z 2026-05-10 17:31:51.744 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-10T17:31:51.744975893Z 2026-05-10 17:31:51.744 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-10T17:31:51.799325158Z 2026-05-10 17:31:51.798 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-10T17:31:52.893747037Z 2026-05-10 17:31:52.893 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-10T17:31:53.137877421Z 2026-05-10 17:31:53.137 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-10T17:31:53.143125707Z 2026-05-10 17:31:53.137 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-10T17:31:53.155774924Z 2026-05-10 17:31:53.155 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-10T17:31:53.162476441Z 2026-05-10 17:31:53.162 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-10T17:31:53.165829555Z 2026-05-10 17:31:53.165 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-10T17:31:53.165903173Z 2026-05-10 17:31:53.165 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-10T17:31:53.202082355Z 2026-05-10 17:31:53.201 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-10T17:31:53.210030516Z 2026-05-10 17:31:53.209 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-10T17:31:53.239479441Z 2026-05-10 17:31:53.238 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-10T17:31:53.252055621Z 2026-05-10 17:31:53.251 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-10T17:31:53.264446725Z 2026-05-10 17:31:53.264 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-10T17:31:53.276252086Z 2026-05-10 17:31:53.275 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-10T17:31:53.357568474Z 2026-05-10 17:31:53.357 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-10T17:31:53.425184578Z 2026-05-10 17:31:53.424 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-10T17:31:53.708760606Z 2026-05-10 17:31:53.708 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-10T17:31:53.947323994Z 2026-05-10 17:31:53.946 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-10T17:31:54.229584138Z 2026-05-10 17:31:54.229 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-10T17:31:54.359080753Z 2026-05-10 17:31:54.358 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-10T17:31:54.365690559Z 2026-05-10 17:31:54.365 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_10.request.log
2026-05-10T17:31:54.369350840Z 2026-05-10 17:31:54.368 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-10T17:31:54.370174493Z 2026-05-10 17:31:54.369 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4978ms
2026-05-10T17:31:54.577462295Z 2026-05-10 17:31:54.576 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=48
2026-05-10T17:31:54.678970773Z 2026-05-10 17:31:54.678 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-10T17:31:59.604596720Z 2026-05-10 17:31:59.603 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-10T17:31:59.653889301Z 2026-05-10 17:31:59.653 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-10T17:31:59.723976317Z 2026-05-10 17:31:59.723 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-10T17:32:00.163127136Z 2026-05-10 17:32:00.162 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-10T17:32:00.163194650Z 2026-05-10 17:32:00.162 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-10T17:32:00.194358432Z 2026-05-10 17:32:00.193 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-10T17:32:00.950352404Z 2026-05-10 17:32:00.949 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-10T17:32:00.971422856Z 2026-05-10 17:32:00.971 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-10T17:32:00.992067716Z 2026-05-10 17:32:00.991 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-10T17:32:00.993305621Z 2026-05-10 17:32:00.992 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-10T17:32:00.993323077Z 2026-05-10 17:32:00.993 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-10T17:32:01.050210688Z 2026-05-10 17:32:01.049 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-10T17:32:01.053391690Z 2026-05-10 17:32:01.053 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-10T17:32:01.055910821Z 2026-05-10 17:32:01.055 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-10T17:32:01.058152940Z 2026-05-10 17:32:01.057 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-10T17:32:01.075871166Z 2026-05-10 17:32:01.075 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-10T17:32:01.083133786Z 2026-05-10 17:32:01.082 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-10T17:32:01.089349017Z 2026-05-10 17:32:01.089 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-10T17:32:01.089372587Z 2026-05-10 17:32:01.089 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-10T17:32:01.095467897Z 2026-05-10 17:32:01.095 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-10T17:32:01.109204974Z 2026-05-10 17:32:01.108 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-10T17:32:01.110702301Z 2026-05-10 17:32:01.110 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-10T17:32:01.129720441Z 2026-05-10 17:32:01.129 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-10T17:32:01.129729821Z 2026-05-10 17:32:01.129 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-10T17:32:01.129783813Z 2026-05-10 17:32:01.129 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-10T17:32:01.132049113Z 2026-05-10 17:32:01.131 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-10T17:32:01.133056820Z 2026-05-10 17:32:01.132 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-10T17:32:01.134219207Z 2026-05-10 17:32:01.133 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72111156159447043-core_node6-n_0000000000
2026-05-10T17:32:01.148013520Z 2026-05-10 17:32:01.147 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-10T17:32:01.267456906Z 2026-05-10 17:32:01.266 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 220] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:01.271916005Z 2026-05-10 17:32:01.271 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-10T17:32:01.283079488Z 2026-05-10 17:32:01.282 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1680
2026-05-10T17:32:01.386998547Z 2026-05-10 17:32:01.386 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:02.552910523Z 2026-05-10 17:32:02.552 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:02.663448729Z 2026-05-10 17:32:02.663 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3183365,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-10T17:31:48.850839292Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-10T17:31:49.375497576Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-10T17:31:49.375520681Z It should be set to 65000 to avoid operational disruption.
2026-05-10T17:31:49.375523655Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-10T17:31:49.375816622Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-10T17:31:49.375825194Z
2026-05-10T17:31:49.384624421Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-10T17:31:49.393225603Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-10T17:31:49.471618283Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-10T17:31:49.506715671Z WARNING: A command line option has enabled the Security Manager
2026-05-10T17:31:49.506734019Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-10T17:31:50.829810623Z 2026-05-10 17:31:50.826 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-10T17:31:51.232612426Z 2026-05-10 17:31:51.232 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-10T17:31:51.266454526Z 2026-05-10 17:31:51.266 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-10T17:31:51.275775429Z 2026-05-10 17:31:51.275 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-10T17:31:51.275846184Z 2026-05-10 17:31:51.275 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-10T17:31:51.275964587Z 2026-05-10 17:31:51.275 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-10T17:31:51.276077618Z 2026-05-10 17:31:51.275 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-10T17:31:51.275590401Z
2026-05-10T17:31:51.279497247Z 2026-05-10 17:31:51.278 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-10T17:31:51.313394735Z 2026-05-10 17:31:51.312 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-10T17:31:51.359316001Z 2026-05-10 17:31:51.358 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-10T17:31:51.609713867Z 2026-05-10 17:31:51.609 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-10T17:31:51.631424796Z 2026-05-10 17:31:51.630 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-10T17:31:51.631481161Z 2026-05-10 17:31:51.631 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-10T17:31:51.631658109Z 2026-05-10 17:31:51.631 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-10T17:31:51.744314690Z 2026-05-10 17:31:51.743 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-10T17:31:51.744438684Z 2026-05-10 17:31:51.744 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-10T17:31:51.744559004Z 2026-05-10 17:31:51.744 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-10T17:31:51.799320091Z 2026-05-10 17:31:51.798 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-10T17:31:53.097843931Z 2026-05-10 17:31:53.097 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-10T17:31:53.300627865Z 2026-05-10 17:31:53.300 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-10T17:31:53.302821417Z 2026-05-10 17:31:53.300 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-10T17:31:53.315942657Z 2026-05-10 17:31:53.315 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-10T17:31:53.323418778Z 2026-05-10 17:31:53.322 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-10T17:31:53.326635519Z 2026-05-10 17:31:53.326 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-10T17:31:53.326654305Z 2026-05-10 17:31:53.326 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-10T17:31:53.362975513Z 2026-05-10 17:31:53.362 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-10T17:31:53.370719189Z 2026-05-10 17:31:53.370 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-10T17:31:53.394030139Z 2026-05-10 17:31:53.393 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-10T17:31:53.406681743Z 2026-05-10 17:31:53.406 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-10T17:31:53.417051263Z 2026-05-10 17:31:53.416 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-10T17:31:53.425825876Z 2026-05-10 17:31:53.425 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-10T17:31:53.485052288Z 2026-05-10 17:31:53.484 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-10T17:31:53.765224053Z 2026-05-10 17:31:53.764 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-10T17:31:53.966981112Z 2026-05-10 17:31:53.966 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-10T17:31:54.229384753Z 2026-05-10 17:31:54.228 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-10T17:31:54.358950803Z 2026-05-10 17:31:54.358 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-10T17:31:54.365735671Z 2026-05-10 17:31:54.365 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_10.request.log
2026-05-10T17:31:54.369179724Z 2026-05-10 17:31:54.368 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-10T17:31:54.370109907Z 2026-05-10 17:31:54.369 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4978ms
2026-05-10T17:31:56.947230708Z 2026-05-10 17:31:56.937 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=99
2026-05-10T17:31:57.021392431Z 2026-05-10 17:31:57.020 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-05-10T17:31:59.163006153Z 2026-05-10 17:31:59.162 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=13
2026-05-10T17:31:59.607349045Z 2026-05-10 17:31:59.606 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-10T17:31:59.665087348Z 2026-05-10 17:31:59.664 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-10T17:31:59.739455510Z 2026-05-10 17:31:59.738 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-10T17:32:00.169069792Z 2026-05-10 17:32:00.168 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-10T17:32:00.169140268Z 2026-05-10 17:32:00.168 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-10T17:32:00.194348575Z 2026-05-10 17:32:00.193 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-10T17:32:00.951992058Z 2026-05-10 17:32:00.951 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-10T17:32:00.972075694Z 2026-05-10 17:32:00.971 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-10T17:32:00.992156624Z 2026-05-10 17:32:00.991 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-10T17:32:00.993243568Z 2026-05-10 17:32:00.992 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-10T17:32:00.993398489Z 2026-05-10 17:32:00.993 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-10T17:32:01.050285397Z 2026-05-10 17:32:01.049 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-10T17:32:01.053373752Z 2026-05-10 17:32:01.053 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-10T17:32:01.055519294Z 2026-05-10 17:32:01.055 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-10T17:32:01.057668903Z 2026-05-10 17:32:01.057 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-10T17:32:01.075824472Z 2026-05-10 17:32:01.075 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-10T17:32:01.083756074Z 2026-05-10 17:32:01.083 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-10T17:32:01.091197544Z 2026-05-10 17:32:01.090 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-10T17:32:01.091252321Z 2026-05-10 17:32:01.090 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-10T17:32:01.098660964Z 2026-05-10 17:32:01.098 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-10T17:32:01.109210936Z 2026-05-10 17:32:01.108 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-10T17:32:01.110906882Z 2026-05-10 17:32:01.110 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-10T17:32:01.129673366Z 2026-05-10 17:32:01.129 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-10T17:32:01.129739001Z 2026-05-10 17:32:01.129 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-10T17:32:01.129741641Z 2026-05-10 17:32:01.129 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-10T17:32:01.131547127Z 2026-05-10 17:32:01.131 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-10T17:32:01.131960744Z 2026-05-10 17:32:01.131 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-10T17:32:01.133154509Z 2026-05-10 17:32:01.132 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72111156159447044-core_node5-n_0000000000
2026-05-10T17:32:01.146781060Z 2026-05-10 17:32:01.146 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-10T17:32:01.267610926Z 2026-05-10 17:32:01.267 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 220] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:01.272918603Z 2026-05-10 17:32:01.272 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-10T17:32:01.283257733Z 2026-05-10 17:32:01.282 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1677
2026-05-10T17:32:01.387145386Z 2026-05-10 17:32:01.386 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:02.552957699Z 2026-05-10 17:32:02.552 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:02.577340384Z 2026-05-10 17:32:02.576 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-10T17:32:02.663522353Z 2026-05-10 17:32:02.663 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-10T17:32:02.666209596Z 2026-05-10 17:32:02.665 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3487
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3183522,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-10T17:31:49.053305274Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 32119 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# 87f3bb8db2f4: Already exists
# 7c0b62ebdf15: Already exists
# 5b95479e5f9b: Already exists
# c913227527c9: Already exists
# 021b66138a10: Pulling fs layer
# 46d35df4948f: Pulling fs layer
# 38dfc96ae577: Pulling fs layer
# 3e8afb22de9f: Pulling fs layer
# 9f335245facc: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 4f4fb700ef54: Waiting
# 3e8afb22de9f: Waiting
# 9f335245facc: Waiting
# 46d35df4948f: Download complete
# 38dfc96ae577: Verifying Checksum
# 38dfc96ae577: Download complete
# 3e8afb22de9f: Verifying Checksum
# 3e8afb22de9f: Download complete
# 9f335245facc: Verifying Checksum
# 9f335245facc: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 021b66138a10: Verifying Checksum
# 021b66138a10: Download complete
# 021b66138a10: Pull complete
# 46d35df4948f: Pull complete
# 38dfc96ae577: Pull complete
# 3e8afb22de9f: Pull complete
# 9f335245facc: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:1a1204a668ec89805462848ba42f5979016249d66019246142b086fa211d4c6f
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# aeb3b44e4745b946048bf62fcd494a7f05f075c5acaa98616f25582e226948ce
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 4cc300ef3f22c810ce11c2ad2b4e81c21d6b1040b77cc1e61f6cbdcd2e898b92
# Starting solr-node2 connected to first node's ZooKeeper
# 3ea56240f87bd4455e16f5dd2b06385f619192eb9cc5000af5c887d8507f9c44
# Started solr-node3 connected to first node's ZooKeeper
# 7755a7fc40a7b7335fd9f6466657ce6d2c00d3a081dc6db9dcd2f197c4b70659
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-10 17:31:58.900; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-10 17:31:58.952; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14496 ms
ok 85 snapshot lifecycle # in 4487 ms
ok 86 snapshot list # in 3144 ms
ok 87 snapshot describe # in 3135 ms
ok 88 start solr with ssl # in 15314 ms
ok 89 use different hostname when not checking peer-name # in 23923 ms
ok 90 start solr with ssl and auth # in 13615 ms
ok 91 start solr with client truststore and security manager # in 15819 ms
ok 92 start solr with mTLS needed # in 44170 ms
ok 93 start solr with mTLS wanted # in 34640 ms
ok 94 test keystore reload # in 56149 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15410 ms
ok 96 stop command for single port # in 17654 ms
ok 97 check stop command doesn't hang # in 204 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7661 ms
ok 99 deprecated system properties converted to modern properties # in 195273 ms
ok 100 start with custom jetty options # in 7859 ms
ok 101 bootstrapping a configset # in 8873 ms
ok 102 status detects locally running solr # in 9079 ms
ok 103 status with --solr-url from user # in 7679 ms
ok 104 status with --port from user # in 7746 ms
ok 105 multiple connection options are prevented # in 805 ms
ok 106 status with invalid --solr-url from user # in 1179 ms
ok 107 status with --short format # in 196058 ms
ok 108 searching solr via locally executed streaming expression # in 1631 ms
ok 109 searching solr via remotely executed streaming expression # in 1187 ms
ok 110 variable interpolation # in 1588 ms
ok 111 searching solr without credentials fails with error # in 1072 ms
ok 112 --version returns Solr version # in 693 ms
ok 113 version as direct tool call still runs # in 706 ms
ok 114 short help # in 744 ms
ok 115 short help is inferred # in 758 ms
ok 116 long help # in 729 ms
ok 117 running subcommands with zk is prevented # in 695 ms
ok 118 listing out files # in 2079 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7450 ms
ok 120 copying files around # in 10567 ms
ok 121 upconfig # in 2478 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1046 ms
ok 123 downconfig # in 1123 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1376 ms
ok 125 env var ZK_HOST is honored # in 3115 ms
ok 126 Cleanup # in 1817 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 56s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/lh2qb32oqshhg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 204 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/204/
No tests ran.
Build Log:
[...truncated 426 lines...]
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:44059/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44059.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44059.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:44059/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c monitors_verbose -s http://localhost:44059 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-09 17:21:22.250; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-09 17:21:22.253; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:44059/solr/admin/collections
ok 82 post solr log into solr via cli # in 8514 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 28 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-09T17:21:47.244765886Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-09T17:21:47.244802289Z It should be set to 65000 to avoid operational disruption.
2026-05-09T17:21:47.244805346Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-09T17:21:47.245035669Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-09T17:21:47.245047400Z
2026-05-09T17:21:47.258139040Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-09T17:21:47.268229031Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-09T17:21:47.315809982Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-09T17:21:47.359332362Z WARNING: A command line option has enabled the Security Manager
2026-05-09T17:21:47.359355032Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-09T17:21:49.901596423Z 2026-05-09 17:21:49.898 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-09T17:21:50.362391813Z 2026-05-09 17:21:50.361 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-09T17:21:50.470469106Z 2026-05-09 17:21:50.470 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-09T17:21:50.501158118Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-09T17:21:50.501217962Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-09T17:21:50.501331085Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-09T17:21:50.501439724Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-09T17:21:50.500878884Z
2026-05-09T17:21:50.504718090Z 2026-05-09 17:21:50.504 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-09T17:21:50.550331830Z 2026-05-09 17:21:50.549 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-09T17:21:50.648063645Z 2026-05-09 17:21:50.647 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-09T17:21:50.648150473Z 2026-05-09 17:21:50.647 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-09T17:21:50.750226046Z 2026-05-09 17:21:50.749 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-09T17:21:52.290944355Z 2026-05-09 17:21:52.289 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-09T17:21:52.778341899Z 2026-05-09 17:21:52.777 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-09T17:21:52.793206897Z 2026-05-09 17:21:52.792 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-09T17:21:52.793235859Z 2026-05-09 17:21:52.792 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-09T17:21:52.798239849Z 2026-05-09 17:21:52.797 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-09T17:21:52.798295826Z 2026-05-09 17:21:52.797 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-09T17:21:52.876122153Z 2026-05-09 17:21:52.875 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-09T17:21:53.300511441Z 2026-05-09 17:21:53.299 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-09T17:21:53.374559304Z 2026-05-09 17:21:53.374 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-09T17:21:53.428242749Z 2026-05-09 17:21:53.427 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-09T17:21:53.451520468Z 2026-05-09 17:21:53.451 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-09T17:21:53.451604890Z 2026-05-09 17:21:53.451 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-09T17:21:53.584349939Z 2026-05-09 17:21:53.583 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-09T17:21:53.617629378Z 2026-05-09 17:21:53.617 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-09T17:21:53.627964351Z 2026-05-09 17:21:53.627 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-09T17:21:53.634917471Z 2026-05-09 17:21:53.634 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72105455046819840-solr-node1:8983_solr-n_0000000000) starting
2026-05-09T17:21:53.722813210Z 2026-05-09 17:21:53.722 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-09T17:21:53.727311567Z 2026-05-09 17:21:53.726 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-09T17:21:53.738353427Z 2026-05-09 17:21:53.737 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-09T17:21:53.885898980Z 2026-05-09 17:21:53.885 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-09T17:21:54.508831382Z 2026-05-09 17:21:54.508 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-09T17:21:54.809808876Z 2026-05-09 17:21:54.809 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-09T17:21:55.181052912Z 2026-05-09 17:21:55.180 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-09T17:21:55.657918531Z 2026-05-09 17:21:55.657 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-09T17:21:55.844313993Z 2026-05-09 17:21:55.843 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_09.request.log
2026-05-09T17:21:55.853538527Z 2026-05-09 17:21:55.852 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-09T17:21:55.856850402Z 2026-05-09 17:21:55.856 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @8589ms
2026-05-09T17:21:57.179331055Z 2026-05-09 17:21:57.178 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=143
2026-05-09T17:21:57.360482862Z 2026-05-09 17:21:57.359 INFO (qtp985324122-97-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=48
2026-05-09T17:22:02.496205229Z 2026-05-09 17:22:02.495 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-09T17:22:02.595294073Z 2026-05-09 17:22:02.594 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-09T17:22:09.214213030Z 2026-05-09 17:22:09.213 INFO (qtp985324122-100-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=18
2026-05-09T17:22:09.275869232Z 2026-05-09 17:22:09.275 INFO (qtp985324122-100-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=15
2026-05-09T17:22:09.303532787Z 2026-05-09 17:22:09.303 INFO (qtp985324122-99-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1
2026-05-09T17:22:09.700935238Z 2026-05-09 17:22:09.700 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=22
2026-05-09T17:22:09.755049630Z 2026-05-09 17:22:09.754 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-09T17:22:09.799425504Z 2026-05-09 17:22:09.798 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-09T17:22:09.822441446Z 2026-05-09 17:22:09.822 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:09.924633700Z 2026-05-09 17:22:09.924 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-09T17:22:09.924651689Z "core":"test-collection_shard3_replica_n1",
2026-05-09T17:22:09.924654934Z "node_name":"solr-node1:8983_solr",
2026-05-09T17:22:09.924657628Z "base_url":"http://solr-node1:8983/solr",
2026-05-09T17:22:09.924660204Z "collection":"test-collection",
2026-05-09T17:22:09.924662817Z "shard":"shard3",
2026-05-09T17:22:09.924665434Z "state":"down",
2026-05-09T17:22:09.924667873Z "type":"NRT",
2026-05-09T17:22:09.924670455Z "operation":"ADDREPLICA",
2026-05-09T17:22:09.924673058Z "waitForFinalState":"false"}
2026-05-09T17:22:09.933014815Z 2026-05-09 17:22:09.932 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-09T17:22:09.933033594Z "core":"test-collection_shard2_replica_n2",
2026-05-09T17:22:09.933037071Z "node_name":"solr-node3:8985_solr",
2026-05-09T17:22:09.933039891Z "base_url":"http://solr-node3:8985/solr",
2026-05-09T17:22:09.933042480Z "collection":"test-collection",
2026-05-09T17:22:09.933049785Z "shard":"shard2",
2026-05-09T17:22:09.933052343Z "state":"down",
2026-05-09T17:22:09.933055002Z "type":"NRT",
2026-05-09T17:22:09.933057461Z "operation":"ADDREPLICA",
2026-05-09T17:22:09.933060216Z "waitForFinalState":"false"}
2026-05-09T17:22:09.940944635Z 2026-05-09 17:22:09.940 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-09T17:22:09.940970691Z "core":"test-collection_shard1_replica_n4",
2026-05-09T17:22:09.940973543Z "node_name":"solr-node2:8984_solr",
2026-05-09T17:22:09.940975671Z "base_url":"http://solr-node2:8984/solr",
2026-05-09T17:22:09.940978035Z "collection":"test-collection",
2026-05-09T17:22:09.940980084Z "shard":"shard1",
2026-05-09T17:22:09.940982011Z "state":"down",
2026-05-09T17:22:09.940984053Z "type":"NRT",
2026-05-09T17:22:09.940985920Z "operation":"ADDREPLICA",
2026-05-09T17:22:09.940988022Z "waitForFinalState":"false"}
2026-05-09T17:22:10.058789426Z 2026-05-09 17:22:10.057 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:10.249220305Z 2026-05-09 17:22:10.248 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-09T17:22:10.374353582Z 2026-05-09 17:22:10.373 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-09T17:22:10.498229447Z 2026-05-09 17:22:10.497 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-09T17:22:11.377438544Z 2026-05-09 17:22:11.376 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-09T17:22:11.377462198Z 2026-05-09 17:22:11.377 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-09T17:22:11.494632323Z 2026-05-09 17:22:11.494 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-09T17:22:13.671191366Z 2026-05-09 17:22:13.670 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-09T17:22:13.754958721Z 2026-05-09 17:22:13.754 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-09T17:22:13.759261563Z 2026-05-09 17:22:13.758 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-09T17:22:13.760153565Z 2026-05-09 17:22:13.759 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-09T17:22:13.760205775Z 2026-05-09 17:22:13.759 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-09T17:22:13.881926253Z 2026-05-09 17:22:13.881 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-09T17:22:13.885151060Z 2026-05-09 17:22:13.884 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-09T17:22:13.889028810Z 2026-05-09 17:22:13.888 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-09T17:22:13.891950147Z 2026-05-09 17:22:13.891 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-09T17:22:13.919231474Z 2026-05-09 17:22:13.918 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-09T17:22:13.970197424Z 2026-05-09 17:22:13.969 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-09T17:22:13.983423312Z 2026-05-09 17:22:13.982 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-09T17:22:13.983441374Z 2026-05-09 17:22:13.983 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-09T17:22:14.003281728Z 2026-05-09 17:22:14.002 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-09T17:22:14.025142229Z 2026-05-09 17:22:14.024 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-09T17:22:14.030222647Z 2026-05-09 17:22:14.029 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-09T17:22:14.044435410Z 2026-05-09 17:22:14.044 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-09T17:22:14.044474457Z 2026-05-09 17:22:14.044 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-09T17:22:14.044538176Z 2026-05-09 17:22:14.044 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-09T17:22:14.045428560Z 2026-05-09 17:22:14.045 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-09T17:22:14.045866878Z 2026-05-09 17:22:14.045 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-09T17:22:14.046970175Z 2026-05-09 17:22:14.046 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72105455046819840-core_node3-n_0000000000
2026-05-09T17:22:14.052627241Z 2026-05-09 17:22:14.052 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-09T17:22:14.159630856Z 2026-05-09 17:22:14.159 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:14.162238559Z 2026-05-09 17:22:14.161 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-09T17:22:14.168256751Z 2026-05-09 17:22:14.168 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3922
2026-05-09T17:22:14.185166181Z 2026-05-09 17:22:14.184 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-09T17:22:14.268265079Z 2026-05-09 17:22:14.267 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:14.270219505Z 2026-05-09 17:22:14.269 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=4550
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2808717,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-09T17:21:46.209323555Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-09T17:21:58.060767412Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-09T17:21:58.060813827Z It should be set to 65000 to avoid operational disruption.
2026-05-09T17:21:58.060817074Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-09T17:21:58.061101957Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-09T17:21:58.061120961Z
2026-05-09T17:21:58.072948163Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-09T17:21:58.082923109Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-09T17:21:58.143784679Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-09T17:21:58.182580349Z WARNING: A command line option has enabled the Security Manager
2026-05-09T17:21:58.182613161Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-09T17:21:59.862456637Z 2026-05-09 17:21:59.858 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-09T17:22:00.358957284Z 2026-05-09 17:22:00.358 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-09T17:22:00.401632248Z 2026-05-09 17:22:00.401 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-09T17:22:00.414823657Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-09T17:22:00.414842397Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-09T17:22:00.414845434Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-09T17:22:00.414848525Z 2026-05-09 17:22:00.414 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-09T17:22:00.414058605Z
2026-05-09T17:22:00.419249204Z 2026-05-09 17:22:00.418 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-09T17:22:00.463447684Z 2026-05-09 17:22:00.462 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-09T17:22:00.518855749Z 2026-05-09 17:22:00.518 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-09T17:22:00.849167268Z 2026-05-09 17:22:00.848 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-09T17:22:00.883506769Z 2026-05-09 17:22:00.882 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-09T17:22:00.883516815Z 2026-05-09 17:22:00.882 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-09T17:22:00.883519418Z 2026-05-09 17:22:00.882 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-09T17:22:01.004061718Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-09T17:22:01.004163033Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-09T17:22:01.004222727Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-09T17:22:01.067746189Z 2026-05-09 17:22:01.067 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-09T17:22:02.347119939Z 2026-05-09 17:22:02.346 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-09T17:22:02.481789040Z 2026-05-09 17:22:02.481 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-09T17:22:02.483147949Z 2026-05-09 17:22:02.481 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-09T17:22:02.491934626Z 2026-05-09 17:22:02.491 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-09T17:22:02.496991865Z 2026-05-09 17:22:02.496 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-09T17:22:02.499917452Z 2026-05-09 17:22:02.499 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-09T17:22:02.499982098Z 2026-05-09 17:22:02.499 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-09T17:22:02.541638890Z 2026-05-09 17:22:02.541 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-09T17:22:02.549508662Z 2026-05-09 17:22:02.548 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-09T17:22:02.569126235Z 2026-05-09 17:22:02.568 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-09T17:22:02.579462985Z 2026-05-09 17:22:02.579 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-09T17:22:02.588240073Z 2026-05-09 17:22:02.587 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-09T17:22:02.596370961Z 2026-05-09 17:22:02.596 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-09T17:22:02.642227955Z 2026-05-09 17:22:02.641 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-09T17:22:02.922779848Z 2026-05-09 17:22:02.922 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-09T17:22:03.171206540Z 2026-05-09 17:22:03.170 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-09T17:22:03.445247590Z 2026-05-09 17:22:03.444 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-09T17:22:03.585783261Z 2026-05-09 17:22:03.585 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-09T17:22:03.593820080Z 2026-05-09 17:22:03.592 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_09.request.log
2026-05-09T17:22:03.598875059Z 2026-05-09 17:22:03.597 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-09T17:22:03.599040532Z 2026-05-09 17:22:03.598 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5517ms
2026-05-09T17:22:04.529866050Z 2026-05-09 17:22:04.529 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-05-09T17:22:04.670682234Z 2026-05-09 17:22:04.669 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-05-09T17:22:10.241786615Z 2026-05-09 17:22:10.241 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-09T17:22:10.301501086Z 2026-05-09 17:22:10.301 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-09T17:22:10.385924678Z 2026-05-09 17:22:10.385 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-09T17:22:10.835084962Z 2026-05-09 17:22:10.834 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-09T17:22:10.835133792Z 2026-05-09 17:22:10.834 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-09T17:22:10.880478773Z 2026-05-09 17:22:10.879 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-09T17:22:11.836884826Z 2026-05-09 17:22:11.836 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-09T17:22:11.860615369Z 2026-05-09 17:22:11.860 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-09T17:22:11.875909087Z 2026-05-09 17:22:11.875 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-09T17:22:11.876911037Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-09T17:22:11.876966403Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-09T17:22:11.933485189Z 2026-05-09 17:22:11.933 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-09T17:22:11.936830690Z 2026-05-09 17:22:11.936 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-09T17:22:11.938775110Z 2026-05-09 17:22:11.938 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-09T17:22:11.940999492Z 2026-05-09 17:22:11.940 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-09T17:22:11.958151033Z 2026-05-09 17:22:11.957 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-09T17:22:11.965536461Z 2026-05-09 17:22:11.965 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-09T17:22:11.972141520Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-09T17:22:11.972186539Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-09T17:22:11.978693399Z 2026-05-09 17:22:11.978 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-09T17:22:11.998497949Z 2026-05-09 17:22:11.997 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-09T17:22:11.999746354Z 2026-05-09 17:22:11.999 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-09T17:22:12.042878887Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-09T17:22:12.042939115Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-09T17:22:12.042984822Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-09T17:22:12.044105160Z 2026-05-09 17:22:12.043 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-09T17:22:12.045322041Z 2026-05-09 17:22:12.045 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-09T17:22:12.046368083Z 2026-05-09 17:22:12.046 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72105455046819844-core_node6-n_0000000000
2026-05-09T17:22:12.080886064Z 2026-05-09 17:22:12.080 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-09T17:22:12.188577723Z 2026-05-09 17:22:12.188 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:12.193125083Z 2026-05-09 17:22:12.192 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-09T17:22:12.219068820Z 2026-05-09 17:22:12.218 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1978
2026-05-09T17:22:12.320889773Z 2026-05-09 17:22:12.320 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:14.160112346Z 2026-05-09 17:22:14.159 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:14.268402126Z 2026-05-09 17:22:14.267 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2809130,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-09T17:21:57.528714979Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-09T17:21:58.062026244Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-09T17:21:58.062050171Z It should be set to 65000 to avoid operational disruption.
2026-05-09T17:21:58.062052873Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-09T17:21:58.062427747Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-09T17:21:58.062444526Z
2026-05-09T17:21:58.072348444Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-09T17:21:58.083107812Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-09T17:21:58.144202222Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-09T17:21:58.182626741Z WARNING: A command line option has enabled the Security Manager
2026-05-09T17:21:58.182655288Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-09T17:21:59.862692123Z 2026-05-09 17:21:59.858 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-09T17:22:00.362091778Z 2026-05-09 17:22:00.361 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-09T17:22:00.401457539Z 2026-05-09 17:22:00.400 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-09T17:22:00.414608273Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-09T17:22:00.414641262Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-09T17:22:00.414831539Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-09T17:22:00.414947708Z 2026-05-09 17:22:00.414 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-09T17:22:00.414052416Z
2026-05-09T17:22:00.419220716Z 2026-05-09 17:22:00.418 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-09T17:22:00.463517772Z 2026-05-09 17:22:00.462 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-09T17:22:00.518855837Z 2026-05-09 17:22:00.518 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-09T17:22:00.849224760Z 2026-05-09 17:22:00.848 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-09T17:22:00.883460686Z 2026-05-09 17:22:00.882 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-09T17:22:00.883653909Z 2026-05-09 17:22:00.882 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-09T17:22:00.883807066Z 2026-05-09 17:22:00.883 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-09T17:22:01.004064279Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-09T17:22:01.004207476Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-09T17:22:01.004218491Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-09T17:22:01.067600864Z 2026-05-09 17:22:01.067 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-09T17:22:02.143629889Z 2026-05-09 17:22:02.143 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-09T17:22:02.355722186Z 2026-05-09 17:22:02.355 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-09T17:22:02.357996220Z 2026-05-09 17:22:02.355 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-09T17:22:02.370797979Z 2026-05-09 17:22:02.370 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-09T17:22:02.379675267Z 2026-05-09 17:22:02.379 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-09T17:22:02.382428548Z 2026-05-09 17:22:02.382 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-09T17:22:02.382535488Z 2026-05-09 17:22:02.382 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-09T17:22:02.421336307Z 2026-05-09 17:22:02.420 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-09T17:22:02.433914003Z 2026-05-09 17:22:02.433 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-09T17:22:02.461270247Z 2026-05-09 17:22:02.460 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-09T17:22:02.472461808Z 2026-05-09 17:22:02.472 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-09T17:22:02.484536471Z 2026-05-09 17:22:02.484 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-09T17:22:02.497433202Z 2026-05-09 17:22:02.497 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-09T17:22:02.558465149Z 2026-05-09 17:22:02.557 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-09T17:22:02.595658418Z 2026-05-09 17:22:02.595 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-09T17:22:02.922738025Z 2026-05-09 17:22:02.922 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-09T17:22:03.170810279Z 2026-05-09 17:22:03.170 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-09T17:22:03.445179630Z 2026-05-09 17:22:03.444 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-09T17:22:03.585814153Z 2026-05-09 17:22:03.585 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-09T17:22:03.593232263Z 2026-05-09 17:22:03.592 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_09.request.log
2026-05-09T17:22:03.596834927Z 2026-05-09 17:22:03.596 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-09T17:22:03.598606796Z 2026-05-09 17:22:03.598 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5517ms
2026-05-09T17:22:07.280438000Z 2026-05-09 17:22:07.272 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=65
2026-05-09T17:22:07.338440988Z 2026-05-09 17:22:07.338 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-05-09T17:22:10.245320031Z 2026-05-09 17:22:10.243 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-09T17:22:10.310449013Z 2026-05-09 17:22:10.310 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-09T17:22:10.428727648Z 2026-05-09 17:22:10.428 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-09T17:22:10.838315333Z 2026-05-09 17:22:10.837 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-09T17:22:10.838368004Z 2026-05-09 17:22:10.838 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-09T17:22:10.880438713Z 2026-05-09 17:22:10.879 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-09T17:22:11.836796217Z 2026-05-09 17:22:11.836 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-09T17:22:11.860577028Z 2026-05-09 17:22:11.860 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-09T17:22:11.875874490Z 2026-05-09 17:22:11.875 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-09T17:22:11.876854821Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-09T17:22:11.876928625Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-09T17:22:11.932877912Z 2026-05-09 17:22:11.932 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-09T17:22:11.935445774Z 2026-05-09 17:22:11.935 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-09T17:22:11.937485115Z 2026-05-09 17:22:11.937 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-09T17:22:11.939799676Z 2026-05-09 17:22:11.939 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-09T17:22:11.958163044Z 2026-05-09 17:22:11.957 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-09T17:22:11.965266816Z 2026-05-09 17:22:11.964 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-09T17:22:11.971593500Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-09T17:22:11.971605961Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-09T17:22:11.978869761Z 2026-05-09 17:22:11.978 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-09T17:22:11.998346115Z 2026-05-09 17:22:11.997 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-09T17:22:11.999790603Z 2026-05-09 17:22:11.999 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-09T17:22:12.042478579Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-09T17:22:12.042501228Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-09T17:22:12.042559992Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-09T17:22:12.043653267Z 2026-05-09 17:22:12.043 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-09T17:22:12.044139018Z 2026-05-09 17:22:12.043 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-09T17:22:12.045563472Z 2026-05-09 17:22:12.045 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72105455046819843-core_node5-n_0000000000
2026-05-09T17:22:12.070727527Z 2026-05-09 17:22:12.070 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-09T17:22:12.188479453Z 2026-05-09 17:22:12.187 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:12.193132508Z 2026-05-09 17:22:12.192 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-09T17:22:12.219125609Z 2026-05-09 17:22:12.218 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1976
2026-05-09T17:22:12.321329653Z 2026-05-09 17:22:12.320 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:14.159900809Z 2026-05-09 17:22:14.159 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-09T17:22:14.268353550Z 2026-05-09 17:22:14.267 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2809258,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-09T17:21:57.746182154Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 35275 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# 87f3bb8db2f4: Already exists
# 7c0b62ebdf15: Already exists
# 5b95479e5f9b: Already exists
# c913227527c9: Already exists
# 7405f766627f: Pulling fs layer
# 32c4816d11cd: Pulling fs layer
# d9e949a1f2d6: Pulling fs layer
# 996d9212abb2: Pulling fs layer
# 5d0c482f99f0: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 996d9212abb2: Waiting
# 4f4fb700ef54: Waiting
# 5d0c482f99f0: Waiting
# d9e949a1f2d6: Verifying Checksum
# d9e949a1f2d6: Download complete
# 32c4816d11cd: Verifying Checksum
# 32c4816d11cd: Download complete
# 996d9212abb2: Verifying Checksum
# 996d9212abb2: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 5d0c482f99f0: Verifying Checksum
# 5d0c482f99f0: Download complete
# 7405f766627f: Verifying Checksum
# 7405f766627f: Download complete
# 7405f766627f: Pull complete
# 32c4816d11cd: Pull complete
# d9e949a1f2d6: Pull complete
# 996d9212abb2: Pull complete
# 5d0c482f99f0: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:d6c725a06cee8371811b8e37f896f7d170274f5190cfe455e37f141e463eb072
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 70788eaa4adf3b34d6858f1cc9dd87c247e3c985d4448e35589c939a0299b364
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 01761ee5c7015a053c8d2654bed84691d439dfd407475ff5e331463c97f35a91
# Starting solr-node2 connected to first node's ZooKeeper
# 488049f0e6a4072a1c3e826defdd5b49c4177d6a2a310a618ddc3bf63d26b776
# Started solr-node3 connected to first node's ZooKeeper
# 13d9be7eeee6d32274c152caaf150ce17cc5f578226c6ff3367acbfdf0865045
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-09 17:22:09.365; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-09 17:22:09.438; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14529 ms
ok 85 snapshot lifecycle # in 4582 ms
ok 86 snapshot list # in 3121 ms
ok 87 snapshot describe # in 3132 ms
ok 88 start solr with ssl # in 15540 ms
ok 89 use different hostname when not checking peer-name # in 23621 ms
ok 90 start solr with ssl and auth # in 13876 ms
ok 91 start solr with client truststore and security manager # in 15748 ms
ok 92 start solr with mTLS needed # in 44820 ms
ok 93 start solr with mTLS wanted # in 34898 ms
ok 94 test keystore reload # in 55966 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15459 ms
ok 96 stop command for single port # in 17755 ms
ok 97 check stop command doesn't hang # in 205 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7766 ms
ok 99 deprecated system properties converted to modern properties # in 195244 ms
ok 100 start with custom jetty options # in 7982 ms
ok 101 bootstrapping a configset # in 8929 ms
ok 102 status detects locally running solr # in 9064 ms
ok 103 status with --solr-url from user # in 7757 ms
ok 104 status with --port from user # in 7776 ms
ok 105 multiple connection options are prevented # in 807 ms
ok 106 status with invalid --solr-url from user # in 1249 ms
ok 107 status with --short format # in 196160 ms
ok 108 searching solr via locally executed streaming expression # in 1608 ms
ok 109 searching solr via remotely executed streaming expression # in 1202 ms
ok 110 variable interpolation # in 1579 ms
ok 111 searching solr without credentials fails with error # in 1083 ms
ok 112 --version returns Solr version # in 690 ms
ok 113 version as direct tool call still runs # in 687 ms
ok 114 short help # in 721 ms
ok 115 short help is inferred # in 726 ms
ok 116 long help # in 731 ms
ok 117 running subcommands with zk is prevented # in 679 ms
ok 118 listing out files # in 2082 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7427 ms
ok 120 copying files around # in 10671 ms
ok 121 upconfig # in 2516 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1043 ms
ok 123 downconfig # in 1129 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1362 ms
ok 125 env var ZK_HOST is honored # in 3090 ms
ok 126 Cleanup # in 1808 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 33s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/njgt67jbkhsys
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 203 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/203/
No tests ran.
Build Log:
[...truncated 417 lines...]
# ERROR: Server refused connection at: http://localhost:46869/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:46869/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# POSTing stdin to http://localhost:46869/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# COMMITting Solr index changes to http://localhost:46869/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:46869/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:46869/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 81 verbose echo the Solr response # in 1063 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46869.
#
# ERROR: Server refused connection at: http://localhost:46869/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46869.
#
# ERROR: Server refused connection at: http://localhost:46869/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8584 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-08T12:42:29.679221110Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-08T12:42:29.679254117Z It should be set to 65000 to avoid operational disruption.
2026-05-08T12:42:29.679257201Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-08T12:42:29.679259242Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-08T12:42:29.679261440Z
2026-05-08T12:42:29.688024594Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-08T12:42:29.696135720Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-08T12:42:29.735897172Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-08T12:42:29.766221308Z WARNING: A command line option has enabled the Security Manager
2026-05-08T12:42:29.766237983Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-08T12:42:30.834417952Z 2026-05-08 12:42:30.831 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-08T12:42:31.090317973Z 2026-05-08 12:42:31.089 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-08T12:42:31.114156352Z 2026-05-08 12:42:31.113 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-08T12:42:31.142402730Z 2026-05-08 12:42:31.120 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-08T12:42:31.142419743Z 2026-05-08 12:42:31.120 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-08T12:42:31.142422639Z 2026-05-08 12:42:31.120 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-08T12:42:31.142425116Z 2026-05-08 12:42:31.120 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-08T12:42:31.120782434Z
2026-05-08T12:42:31.142427585Z 2026-05-08 12:42:31.123 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-08T12:42:31.145480479Z 2026-05-08 12:42:31.144 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-08T12:42:31.159197168Z 2026-05-08 12:42:31.158 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-08T12:42:31.159212721Z 2026-05-08 12:42:31.158 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-08T12:42:31.194938812Z 2026-05-08 12:42:31.194 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-08T12:42:31.953729068Z 2026-05-08 12:42:31.953 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-08T12:42:32.210351487Z 2026-05-08 12:42:32.209 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-08T12:42:32.217843076Z 2026-05-08 12:42:32.217 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-08T12:42:32.218572806Z 2026-05-08 12:42:32.217 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-08T12:42:32.221311685Z 2026-05-08 12:42:32.220 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-08T12:42:32.221325032Z 2026-05-08 12:42:32.220 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-08T12:42:32.258231946Z 2026-05-08 12:42:32.257 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-08T12:42:32.724086238Z 2026-05-08 12:42:32.723 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-08T12:42:32.736026653Z 2026-05-08 12:42:32.735 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-08T12:42:32.786125983Z 2026-05-08 12:42:32.784 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-08T12:42:32.820810145Z 2026-05-08 12:42:32.819 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-08T12:42:32.820861954Z 2026-05-08 12:42:32.820 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-08T12:42:32.936050264Z 2026-05-08 12:42:32.935 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-08T12:42:32.963967828Z 2026-05-08 12:42:32.963 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-08T12:42:32.979882724Z 2026-05-08 12:42:32.979 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-08T12:42:32.985748464Z 2026-05-08 12:42:32.985 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72098694310920192-solr-node1:8983_solr-n_0000000000) starting
2026-05-08T12:42:33.057340789Z 2026-05-08 12:42:33.055 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-08T12:42:33.058802645Z 2026-05-08 12:42:33.058 INFO (OverseerStateUpdate-72098694310920192-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-08T12:42:33.068641664Z 2026-05-08 12:42:33.068 INFO (OverseerStateUpdate-72098694310920192-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-08T12:42:33.109706859Z 2026-05-08 12:42:33.109 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-08T12:42:33.509562739Z 2026-05-08 12:42:33.508 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-08T12:42:33.696426334Z 2026-05-08 12:42:33.695 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-08T12:42:33.974738388Z 2026-05-08 12:42:33.974 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-08T12:42:34.117299399Z 2026-05-08 12:42:34.116 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-08T12:42:34.125708655Z 2026-05-08 12:42:34.125 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_08.request.log
2026-05-08T12:42:34.131003872Z 2026-05-08 12:42:34.130 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-08T12:42:34.177936820Z 2026-05-08 12:42:34.177 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4482ms
2026-05-08T12:42:34.329824315Z 2026-05-08 12:42:34.329 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=43
2026-05-08T12:42:34.387694932Z 2026-05-08 12:42:34.387 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-08T12:42:37.662683253Z 2026-05-08 12:42:37.662 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-08T12:42:38.054818279Z 2026-05-08 12:42:38.054 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-08T12:42:41.354087424Z 2026-05-08 12:42:41.353 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-05-08T12:42:41.376543200Z 2026-05-08 12:42:41.376 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=7
2026-05-08T12:42:41.393327985Z 2026-05-08 12:42:41.392 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1
2026-05-08T12:42:41.587038116Z 2026-05-08 12:42:41.586 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=5
2026-05-08T12:42:41.619991941Z 2026-05-08 12:42:41.619 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-08T12:42:41.637243866Z 2026-05-08 12:42:41.636 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-08T12:42:41.642929488Z 2026-05-08 12:42:41.642 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:41.680551422Z 2026-05-08 12:42:41.680 INFO (OverseerStateUpdate-72098694310920192-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-08T12:42:41.680569138Z "core":"test-collection_shard3_replica_n1",
2026-05-08T12:42:41.680572426Z "node_name":"solr-node1:8983_solr",
2026-05-08T12:42:41.680575172Z "base_url":"http://solr-node1:8983/solr",
2026-05-08T12:42:41.680577818Z "collection":"test-collection",
2026-05-08T12:42:41.680580463Z "shard":"shard3",
2026-05-08T12:42:41.680582998Z "state":"down",
2026-05-08T12:42:41.680585416Z "type":"NRT",
2026-05-08T12:42:41.680587915Z "operation":"ADDREPLICA",
2026-05-08T12:42:41.680590367Z "waitForFinalState":"false"}
2026-05-08T12:42:41.685352188Z 2026-05-08 12:42:41.684 INFO (OverseerStateUpdate-72098694310920192-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-08T12:42:41.685365072Z "core":"test-collection_shard2_replica_n2",
2026-05-08T12:42:41.685368136Z "node_name":"solr-node3:8985_solr",
2026-05-08T12:42:41.685370812Z "base_url":"http://solr-node3:8985/solr",
2026-05-08T12:42:41.685373448Z "collection":"test-collection",
2026-05-08T12:42:41.685376073Z "shard":"shard2",
2026-05-08T12:42:41.685378787Z "state":"down",
2026-05-08T12:42:41.685381283Z "type":"NRT",
2026-05-08T12:42:41.685383781Z "operation":"ADDREPLICA",
2026-05-08T12:42:41.685386761Z "waitForFinalState":"false"}
2026-05-08T12:42:41.689445325Z 2026-05-08 12:42:41.689 INFO (OverseerStateUpdate-72098694310920192-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-08T12:42:41.689469419Z "core":"test-collection_shard1_replica_n4",
2026-05-08T12:42:41.689472297Z "node_name":"solr-node2:8984_solr",
2026-05-08T12:42:41.689474447Z "base_url":"http://solr-node2:8984/solr",
2026-05-08T12:42:41.689476406Z "collection":"test-collection",
2026-05-08T12:42:41.689478441Z "shard":"shard1",
2026-05-08T12:42:41.689480331Z "state":"down",
2026-05-08T12:42:41.689482502Z "type":"NRT",
2026-05-08T12:42:41.689484560Z "operation":"ADDREPLICA",
2026-05-08T12:42:41.689486563Z "waitForFinalState":"false"}
2026-05-08T12:42:41.796818684Z 2026-05-08 12:42:41.796 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:41.884604776Z 2026-05-08 12:42:41.882 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-08T12:42:41.965540047Z 2026-05-08 12:42:41.963 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-08T12:42:42.012397015Z 2026-05-08 12:42:42.011 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-08T12:42:42.374014410Z 2026-05-08 12:42:42.373 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-08T12:42:42.374036490Z 2026-05-08 12:42:42.373 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-08T12:42:42.400459229Z 2026-05-08 12:42:42.399 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-08T12:42:43.086213395Z 2026-05-08 12:42:43.084 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-08T12:42:43.113929121Z 2026-05-08 12:42:43.113 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-08T12:42:43.119987141Z 2026-05-08 12:42:43.119 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-08T12:42:43.121126727Z 2026-05-08 12:42:43.120 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-08T12:42:43.121136604Z 2026-05-08 12:42:43.120 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-08T12:42:43.183690145Z 2026-05-08 12:42:43.183 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-08T12:42:43.186476534Z 2026-05-08 12:42:43.186 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-08T12:42:43.188181163Z 2026-05-08 12:42:43.187 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-08T12:42:43.189701106Z 2026-05-08 12:42:43.189 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-08T12:42:43.202539386Z 2026-05-08 12:42:43.202 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-08T12:42:43.212993153Z 2026-05-08 12:42:43.212 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-08T12:42:43.237395718Z 2026-05-08 12:42:43.237 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-08T12:42:43.237447642Z 2026-05-08 12:42:43.237 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-08T12:42:43.251030179Z 2026-05-08 12:42:43.250 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-08T12:42:43.264475075Z 2026-05-08 12:42:43.263 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-08T12:42:43.265824912Z 2026-05-08 12:42:43.265 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-08T12:42:43.276314992Z 2026-05-08 12:42:43.276 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-08T12:42:43.276353376Z 2026-05-08 12:42:43.276 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-08T12:42:43.276386408Z 2026-05-08 12:42:43.276 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-08T12:42:43.277575563Z 2026-05-08 12:42:43.277 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-08T12:42:43.278030337Z 2026-05-08 12:42:43.277 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-08T12:42:43.279090651Z 2026-05-08 12:42:43.278 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72098694310920192-core_node3-n_0000000000
2026-05-08T12:42:43.284196734Z 2026-05-08 12:42:43.283 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-08T12:42:43.389387119Z 2026-05-08 12:42:43.388 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.392071300Z 2026-05-08 12:42:43.391 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-08T12:42:43.398502695Z 2026-05-08 12:42:43.398 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1516
2026-05-08T12:42:43.405830908Z 2026-05-08 12:42:43.405 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-08T12:42:43.498110042Z 2026-05-08 12:42:43.497 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.500293954Z 2026-05-08 12:42:43.500 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=1894
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1429117,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-08T12:42:29.450977518Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-08T12:42:34.884590987Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-08T12:42:34.884611363Z It should be set to 65000 to avoid operational disruption.
2026-05-08T12:42:34.884614389Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-08T12:42:34.884938764Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-08T12:42:34.884948031Z
2026-05-08T12:42:34.893579199Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-08T12:42:34.900948657Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-08T12:42:34.947875632Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-08T12:42:34.977895268Z WARNING: A command line option has enabled the Security Manager
2026-05-08T12:42:34.977910749Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-08T12:42:35.998250168Z 2026-05-08 12:42:35.995 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-08T12:42:36.259227719Z 2026-05-08 12:42:36.258 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-08T12:42:36.283692108Z 2026-05-08 12:42:36.283 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-08T12:42:36.290792961Z 2026-05-08 12:42:36.290 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-08T12:42:36.290863695Z 2026-05-08 12:42:36.290 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-08T12:42:36.290906463Z 2026-05-08 12:42:36.290 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-08T12:42:36.291070513Z 2026-05-08 12:42:36.290 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-08T12:42:36.290526651Z
2026-05-08T12:42:36.293908293Z 2026-05-08 12:42:36.293 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-08T12:42:36.315727516Z 2026-05-08 12:42:36.315 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-08T12:42:36.339259051Z 2026-05-08 12:42:36.338 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-08T12:42:36.475743145Z 2026-05-08 12:42:36.474 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-08T12:42:36.495268431Z 2026-05-08 12:42:36.494 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-08T12:42:36.495292701Z 2026-05-08 12:42:36.494 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-08T12:42:36.495295464Z 2026-05-08 12:42:36.494 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-08T12:42:36.604385032Z 2026-05-08 12:42:36.603 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-08T12:42:36.604605223Z 2026-05-08 12:42:36.604 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-08T12:42:36.604698921Z 2026-05-08 12:42:36.604 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-08T12:42:36.636822300Z 2026-05-08 12:42:36.636 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-08T12:42:37.394154803Z 2026-05-08 12:42:37.393 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-08T12:42:37.523884833Z 2026-05-08 12:42:37.523 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-08T12:42:37.525246523Z 2026-05-08 12:42:37.523 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-08T12:42:37.533136284Z 2026-05-08 12:42:37.532 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-08T12:42:37.537625613Z 2026-05-08 12:42:37.537 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-08T12:42:37.541569137Z 2026-05-08 12:42:37.541 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-08T12:42:37.541668330Z 2026-05-08 12:42:37.541 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-08T12:42:37.570468650Z 2026-05-08 12:42:37.569 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-08T12:42:37.578410781Z 2026-05-08 12:42:37.577 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-08T12:42:37.598047551Z 2026-05-08 12:42:37.597 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-08T12:42:37.641150426Z 2026-05-08 12:42:37.640 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-08T12:42:37.652040988Z 2026-05-08 12:42:37.651 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-08T12:42:37.663589337Z 2026-05-08 12:42:37.663 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-08T12:42:37.730380178Z 2026-05-08 12:42:37.729 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-08T12:42:38.003145814Z 2026-05-08 12:42:38.002 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-08T12:42:38.055196563Z 2026-05-08 12:42:38.054 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-08T12:42:38.194175578Z 2026-05-08 12:42:38.193 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-08T12:42:38.453143225Z 2026-05-08 12:42:38.452 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-08T12:42:38.585920707Z 2026-05-08 12:42:38.585 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-08T12:42:38.592372823Z 2026-05-08 12:42:38.591 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_08.request.log
2026-05-08T12:42:38.596613326Z 2026-05-08 12:42:38.596 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-08T12:42:38.598076538Z 2026-05-08 12:42:38.597 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3698ms
2026-05-08T12:42:38.755298383Z 2026-05-08 12:42:38.754 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=44
2026-05-08T12:42:38.809350007Z 2026-05-08 12:42:38.808 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-05-08T12:42:41.882620998Z 2026-05-08 12:42:41.882 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-08T12:42:41.929201923Z 2026-05-08 12:42:41.928 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-08T12:42:41.984016841Z 2026-05-08 12:42:41.983 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-08T12:42:42.265206739Z 2026-05-08 12:42:42.264 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-08T12:42:42.265236586Z 2026-05-08 12:42:42.264 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-08T12:42:42.283805158Z 2026-05-08 12:42:42.283 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-08T12:42:42.728029548Z 2026-05-08 12:42:42.727 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-08T12:42:42.746087739Z 2026-05-08 12:42:42.745 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-08T12:42:42.749456974Z 2026-05-08 12:42:42.749 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-08T12:42:42.750470735Z 2026-05-08 12:42:42.749 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-08T12:42:42.750529365Z 2026-05-08 12:42:42.750 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-08T12:42:42.787532507Z 2026-05-08 12:42:42.787 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-08T12:42:42.790355399Z 2026-05-08 12:42:42.790 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-08T12:42:42.792015249Z 2026-05-08 12:42:42.791 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-08T12:42:42.793403774Z 2026-05-08 12:42:42.793 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-08T12:42:42.803969554Z 2026-05-08 12:42:42.803 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-08T12:42:42.811070352Z 2026-05-08 12:42:42.810 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-08T12:42:42.818456333Z 2026-05-08 12:42:42.816 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-08T12:42:42.818475629Z 2026-05-08 12:42:42.816 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-08T12:42:42.823405593Z 2026-05-08 12:42:42.823 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-08T12:42:42.832999325Z 2026-05-08 12:42:42.832 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-08T12:42:42.834300545Z 2026-05-08 12:42:42.834 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-08T12:42:42.950234615Z 2026-05-08 12:42:42.949 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-08T12:42:42.950257013Z 2026-05-08 12:42:42.949 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-08T12:42:42.950345755Z 2026-05-08 12:42:42.950 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-08T12:42:42.951408211Z 2026-05-08 12:42:42.951 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-08T12:42:42.951874525Z 2026-05-08 12:42:42.951 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-08T12:42:42.952908638Z 2026-05-08 12:42:42.952 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72098694310920195-core_node6-n_0000000000
2026-05-08T12:42:42.961521962Z 2026-05-08 12:42:42.961 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-08T12:42:43.118669714Z 2026-05-08 12:42:43.118 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.122092064Z 2026-05-08 12:42:43.121 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-08T12:42:43.130265852Z 2026-05-08 12:42:43.129 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1249
2026-05-08T12:42:43.234169886Z 2026-05-08 12:42:43.233 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.389597659Z 2026-05-08 12:42:43.389 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.498289455Z 2026-05-08 12:42:43.497 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1429520,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-08T12:42:34.474299525Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-08T12:42:35.097362810Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-08T12:42:35.097395144Z It should be set to 65000 to avoid operational disruption.
2026-05-08T12:42:35.097398271Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-08T12:42:35.098239435Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-08T12:42:35.098247078Z
2026-05-08T12:42:35.106235442Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-08T12:42:35.113639047Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-08T12:42:35.161030952Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-08T12:42:35.190863501Z WARNING: A command line option has enabled the Security Manager
2026-05-08T12:42:35.190884037Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-08T12:42:36.267012969Z 2026-05-08 12:42:36.263 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-08T12:42:36.545703376Z 2026-05-08 12:42:36.545 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-08T12:42:36.570091058Z 2026-05-08 12:42:36.569 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-08T12:42:36.576817383Z 2026-05-08 12:42:36.576 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-08T12:42:36.576873409Z 2026-05-08 12:42:36.576 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-08T12:42:36.576994145Z 2026-05-08 12:42:36.576 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-08T12:42:36.577146877Z 2026-05-08 12:42:36.576 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-08T12:42:36.576558744Z
2026-05-08T12:42:36.579714516Z 2026-05-08 12:42:36.579 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-08T12:42:36.600668787Z 2026-05-08 12:42:36.600 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-08T12:42:36.624292380Z 2026-05-08 12:42:36.623 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-08T12:42:36.761189543Z 2026-05-08 12:42:36.759 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-08T12:42:36.776803251Z 2026-05-08 12:42:36.776 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-08T12:42:36.776903433Z 2026-05-08 12:42:36.776 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-08T12:42:36.777174160Z 2026-05-08 12:42:36.776 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-08T12:42:36.892082779Z 2026-05-08 12:42:36.891 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-08T12:42:36.892129459Z 2026-05-08 12:42:36.891 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-08T12:42:36.892266356Z 2026-05-08 12:42:36.892 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-08T12:42:36.924399123Z 2026-05-08 12:42:36.923 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-08T12:42:37.803869884Z 2026-05-08 12:42:37.803 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-08T12:42:37.938565394Z 2026-05-08 12:42:37.938 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-08T12:42:37.939837543Z 2026-05-08 12:42:37.938 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-08T12:42:37.948149150Z 2026-05-08 12:42:37.947 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-08T12:42:37.952642684Z 2026-05-08 12:42:37.952 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-08T12:42:37.955760325Z 2026-05-08 12:42:37.955 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-08T12:42:37.955859732Z 2026-05-08 12:42:37.955 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-08T12:42:37.984412664Z 2026-05-08 12:42:37.983 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-08T12:42:37.992272052Z 2026-05-08 12:42:37.991 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-08T12:42:38.024904047Z 2026-05-08 12:42:38.024 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-08T12:42:38.037746616Z 2026-05-08 12:42:38.037 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-08T12:42:38.047270642Z 2026-05-08 12:42:38.046 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-08T12:42:38.055681953Z 2026-05-08 12:42:38.055 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-08T12:42:38.095799125Z 2026-05-08 12:42:38.095 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-08T12:42:38.352840387Z 2026-05-08 12:42:38.352 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-08T12:42:38.516942573Z 2026-05-08 12:42:38.516 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-08T12:42:38.762559608Z 2026-05-08 12:42:38.762 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-08T12:42:38.882838177Z 2026-05-08 12:42:38.882 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-08T12:42:38.887896180Z 2026-05-08 12:42:38.887 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_08.request.log
2026-05-08T12:42:38.891294111Z 2026-05-08 12:42:38.890 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-08T12:42:38.892007484Z 2026-05-08 12:42:38.891 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3779ms
2026-05-08T12:42:40.278799234Z 2026-05-08 12:42:40.273 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=75
2026-05-08T12:42:40.338917969Z 2026-05-08 12:42:40.338 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-05-08T12:42:41.884139523Z 2026-05-08 12:42:41.883 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-08T12:42:41.944940689Z 2026-05-08 12:42:41.944 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-08T12:42:42.030833176Z 2026-05-08 12:42:42.030 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-08T12:42:42.313316611Z 2026-05-08 12:42:42.312 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-08T12:42:42.313341511Z 2026-05-08 12:42:42.312 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-08T12:42:42.370142065Z 2026-05-08 12:42:42.368 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-08T12:42:42.839241968Z 2026-05-08 12:42:42.838 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-08T12:42:42.859706745Z 2026-05-08 12:42:42.859 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-08T12:42:42.863008460Z 2026-05-08 12:42:42.862 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-08T12:42:42.864067350Z 2026-05-08 12:42:42.863 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-08T12:42:42.864142047Z 2026-05-08 12:42:42.863 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-08T12:42:42.937633433Z 2026-05-08 12:42:42.937 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-08T12:42:42.940464421Z 2026-05-08 12:42:42.940 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-08T12:42:42.944072469Z 2026-05-08 12:42:42.943 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-08T12:42:42.946078519Z 2026-05-08 12:42:42.945 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-08T12:42:42.958258238Z 2026-05-08 12:42:42.957 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-08T12:42:42.966818695Z 2026-05-08 12:42:42.966 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-08T12:42:42.972140714Z 2026-05-08 12:42:42.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-08T12:42:42.972450722Z 2026-05-08 12:42:42.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-08T12:42:42.979258676Z 2026-05-08 12:42:42.978 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-08T12:42:42.988033559Z 2026-05-08 12:42:42.987 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-08T12:42:42.989280622Z 2026-05-08 12:42:42.989 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-08T12:42:43.000744496Z 2026-05-08 12:42:42.999 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-08T12:42:43.000845816Z 2026-05-08 12:42:43.000 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-08T12:42:43.000855629Z 2026-05-08 12:42:43.000 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-08T12:42:43.002230767Z 2026-05-08 12:42:43.001 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-08T12:42:43.003274000Z 2026-05-08 12:42:43.002 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-08T12:42:43.004236424Z 2026-05-08 12:42:43.004 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72098694310920196-core_node5-n_0000000000
2026-05-08T12:42:43.012274857Z 2026-05-08 12:42:43.011 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-08T12:42:43.118699592Z 2026-05-08 12:42:43.118 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.122065634Z 2026-05-08 12:42:43.121 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-08T12:42:43.130799307Z 2026-05-08 12:42:43.130 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1249
2026-05-08T12:42:43.234327863Z 2026-05-08 12:42:43.233 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.389572947Z 2026-05-08 12:42:43.389 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-08T12:42:43.498265457Z 2026-05-08 12:42:43.497 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1429703,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-08T12:42:34.895128632Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 26851 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 66b76b382631: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# 87f3bb8db2f4: Pulling fs layer
# 7c0b62ebdf15: Pulling fs layer
# 5b95479e5f9b: Pulling fs layer
# c913227527c9: Pulling fs layer
# 7d608e2e5c50: Pulling fs layer
# d096686110d5: Pulling fs layer
# dee0ed9ed5db: Pulling fs layer
# d6fa78b53a89: Pulling fs layer
# 6d7641c18e87: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# d096686110d5: Waiting
# dee0ed9ed5db: Waiting
# 4f4fb700ef54: Waiting
# d6fa78b53a89: Waiting
# 6d7641c18e87: Waiting
# c913227527c9: Waiting
# 7d608e2e5c50: Waiting
# 5b95479e5f9b: Verifying Checksum
# 5b95479e5f9b: Download complete
# 87f3bb8db2f4: Verifying Checksum
# 87f3bb8db2f4: Download complete
# c913227527c9: Verifying Checksum
# c913227527c9: Download complete
# d096686110d5: Download complete
# 7c0b62ebdf15: Verifying Checksum
# 7c0b62ebdf15: Download complete
# dee0ed9ed5db: Verifying Checksum
# dee0ed9ed5db: Download complete
# d6fa78b53a89: Download complete
# 6d7641c18e87: Verifying Checksum
# 6d7641c18e87: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 7d608e2e5c50: Verifying Checksum
# 7d608e2e5c50: Download complete
# 87f3bb8db2f4: Pull complete
# 7c0b62ebdf15: Pull complete
# 5b95479e5f9b: Pull complete
# c913227527c9: Pull complete
# 7d608e2e5c50: Pull complete
# d096686110d5: Pull complete
# dee0ed9ed5db: Pull complete
# d6fa78b53a89: Pull complete
# 6d7641c18e87: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:0ee3a103bf8b6f4e3e5930f57e3afd2838c94257302b01b4419ae0064aeb79dc
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 499f55623adf0bc3c3df13d31577bcfa03dd259c300f260d5ff37425829da841
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 2e7459b1d3e050cdaf96488265e5099e98a9f3008031436cc4c842dfd4910be3
# Starting solr-node2 connected to first node's ZooKeeper
# ca27b3ef21f21fb6e84976c0f0c18148f22d918e7da45e52475902bcc2e9eb22
# Started solr-node3 connected to first node's ZooKeeper
# 01fc858d41785bc3a8e4367f41ba306f967b7a2437cd8e616f8b6313b4eb8c4d
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-08 12:42:41.426; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-08 12:42:41.454; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14451 ms
ok 85 snapshot lifecycle # in 4523 ms
ok 86 snapshot list # in 3174 ms
ok 87 snapshot describe # in 3192 ms
ok 88 start solr with ssl # in 15528 ms
ok 89 use different hostname when not checking peer-name # in 23847 ms
ok 90 start solr with ssl and auth # in 13867 ms
ok 91 start solr with client truststore and security manager # in 15704 ms
ok 92 start solr with mTLS needed # in 44364 ms
ok 93 start solr with mTLS wanted # in 33290 ms
ok 94 test keystore reload # in 56073 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15457 ms
ok 96 stop command for single port # in 17651 ms
ok 97 check stop command doesn't hang # in 211 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7944 ms
ok 99 deprecated system properties converted to modern properties # in 195286 ms
ok 100 start with custom jetty options # in 7828 ms
ok 101 bootstrapping a configset # in 8826 ms
ok 102 status detects locally running solr # in 9290 ms
ok 103 status with --solr-url from user # in 7773 ms
ok 104 status with --port from user # in 7697 ms
ok 105 multiple connection options are prevented # in 841 ms
ok 106 status with invalid --solr-url from user # in 1245 ms
ok 107 status with --short format # in 196020 ms
ok 108 searching solr via locally executed streaming expression # in 1610 ms
ok 109 searching solr via remotely executed streaming expression # in 1191 ms
ok 110 variable interpolation # in 1582 ms
ok 111 searching solr without credentials fails with error # in 1075 ms
ok 112 --version returns Solr version # in 688 ms
ok 113 version as direct tool call still runs # in 684 ms
ok 114 short help # in 721 ms
ok 115 short help is inferred # in 712 ms
ok 116 long help # in 718 ms
ok 117 running subcommands with zk is prevented # in 691 ms
ok 118 listing out files # in 2087 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7462 ms
ok 120 copying files around # in 10574 ms
ok 121 upconfig # in 2478 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1070 ms
ok 123 downconfig # in 1181 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1355 ms
ok 125 env var ZK_HOST is honored # in 3067 ms
ok 126 Cleanup # in 1813 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 34s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/433s72ug26656
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 202 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/202/
No tests ran.
Build Log:
[...truncated 425 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:40993/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:40993/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:40993 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-07 15:33:55.479; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-07 15:33:55.482; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:40993/solr/admin/collections
not ok 81 verbose echo the Solr response # in 1065 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993.
#
# ERROR: Server refused connection at: http://localhost:40993/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993.
#
# ERROR: Server refused connection at: http://localhost:40993/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8788 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 22 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-07T15:34:22.136581636Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-07T15:34:22.136612326Z It should be set to 65000 to avoid operational disruption.
2026-05-07T15:34:22.136615331Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-07T15:34:22.136880080Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-07T15:34:22.136890886Z
2026-05-07T15:34:22.171443068Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-07T15:34:22.179759880Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-07T15:34:22.229819487Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-07T15:34:22.264281706Z WARNING: A command line option has enabled the Security Manager
2026-05-07T15:34:22.264304653Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-07T15:34:24.075907719Z 2026-05-07 15:34:24.072 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-07T15:34:24.374379348Z 2026-05-07 15:34:24.373 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-07T15:34:24.428417968Z 2026-05-07 15:34:24.427 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-07T15:34:24.434805675Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-07T15:34:24.434888245Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-07T15:34:24.435000221Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-07T15:34:24.435126859Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-07T15:34:24.434630016Z
2026-05-07T15:34:24.437572102Z 2026-05-07 15:34:24.437 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-07T15:34:24.458020535Z 2026-05-07 15:34:24.457 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-07T15:34:24.499251326Z 2026-05-07 15:34:24.498 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-07T15:34:24.499286581Z 2026-05-07 15:34:24.498 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-07T15:34:24.636708204Z 2026-05-07 15:34:24.635 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-07T15:34:25.506393075Z 2026-05-07 15:34:25.505 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-07T15:34:25.838075856Z 2026-05-07 15:34:25.837 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-07T15:34:25.859116745Z 2026-05-07 15:34:25.858 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-07T15:34:25.859219130Z 2026-05-07 15:34:25.858 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-07T15:34:25.865196285Z 2026-05-07 15:34:25.864 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-07T15:34:25.865276592Z 2026-05-07 15:34:25.864 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-07T15:34:25.953568020Z 2026-05-07 15:34:25.953 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-07T15:34:26.368901114Z 2026-05-07 15:34:26.368 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-07T15:34:26.382538175Z 2026-05-07 15:34:26.382 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-07T15:34:26.436388667Z 2026-05-07 15:34:26.435 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-07T15:34:26.468350930Z 2026-05-07 15:34:26.467 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-07T15:34:26.468461402Z 2026-05-07 15:34:26.468 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-07T15:34:26.601322202Z 2026-05-07 15:34:26.600 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-07T15:34:26.626818521Z 2026-05-07 15:34:26.626 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-07T15:34:26.640417378Z 2026-05-07 15:34:26.639 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-07T15:34:26.648711261Z 2026-05-07 15:34:26.648 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72093707922833408-solr-node1:8983_solr-n_0000000000) starting
2026-05-07T15:34:26.842939865Z 2026-05-07 15:34:26.842 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-07T15:34:26.844505598Z 2026-05-07 15:34:26.844 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-07T15:34:26.855519149Z 2026-05-07 15:34:26.854 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-07T15:34:26.904784200Z 2026-05-07 15:34:26.904 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-07T15:34:27.432672217Z 2026-05-07 15:34:27.432 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-07T15:34:27.685075571Z 2026-05-07 15:34:27.684 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-07T15:34:28.032954172Z 2026-05-07 15:34:28.032 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-07T15:34:28.331479711Z 2026-05-07 15:34:28.330 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-07T15:34:28.337221569Z 2026-05-07 15:34:28.336 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_07.request.log
2026-05-07T15:34:28.340345620Z 2026-05-07 15:34:28.339 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-07T15:34:28.341131817Z 2026-05-07 15:34:28.340 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @6162ms
2026-05-07T15:34:28.862961119Z 2026-05-07 15:34:28.862 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=48
2026-05-07T15:34:29.088906207Z 2026-05-07 15:34:29.088 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=21
2026-05-07T15:34:32.554017953Z 2026-05-07 15:34:32.553 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-07T15:34:36.208123944Z 2026-05-07 15:34:36.207 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-07T15:34:40.736944593Z 2026-05-07 15:34:40.735 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=76
2026-05-07T15:34:40.780331768Z 2026-05-07 15:34:40.768 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-05-07T15:34:40.794171467Z 2026-05-07 15:34:40.793 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-07T15:34:41.039682345Z 2026-05-07 15:34:41.038 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=28
2026-05-07T15:34:41.092024964Z 2026-05-07 15:34:41.091 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-07T15:34:41.118361480Z 2026-05-07 15:34:41.117 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-07T15:34:41.126431936Z 2026-05-07 15:34:41.126 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:41.180247976Z 2026-05-07 15:34:41.179 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-07T15:34:41.180268495Z "core":"test-collection_shard3_replica_n1",
2026-05-07T15:34:41.180271314Z "node_name":"solr-node1:8983_solr",
2026-05-07T15:34:41.180273445Z "base_url":"http://solr-node1:8983/solr",
2026-05-07T15:34:41.180275439Z "collection":"test-collection",
2026-05-07T15:34:41.180277594Z "shard":"shard3",
2026-05-07T15:34:41.180279626Z "state":"down",
2026-05-07T15:34:41.180281549Z "type":"NRT",
2026-05-07T15:34:41.180283479Z "operation":"ADDREPLICA",
2026-05-07T15:34:41.180285452Z "waitForFinalState":"false"}
2026-05-07T15:34:41.186339396Z 2026-05-07 15:34:41.185 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-07T15:34:41.186356080Z "core":"test-collection_shard2_replica_n2",
2026-05-07T15:34:41.186359286Z "node_name":"solr-node3:8985_solr",
2026-05-07T15:34:41.186361533Z "base_url":"http://solr-node3:8985/solr",
2026-05-07T15:34:41.186363496Z "collection":"test-collection",
2026-05-07T15:34:41.186365573Z "shard":"shard2",
2026-05-07T15:34:41.186367486Z "state":"down",
2026-05-07T15:34:41.186369391Z "type":"NRT",
2026-05-07T15:34:41.186371293Z "operation":"ADDREPLICA",
2026-05-07T15:34:41.186373469Z "waitForFinalState":"false"}
2026-05-07T15:34:41.191755874Z 2026-05-07 15:34:41.191 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-07T15:34:41.191777563Z "core":"test-collection_shard1_replica_n4",
2026-05-07T15:34:41.191780349Z "node_name":"solr-node2:8984_solr",
2026-05-07T15:34:41.191782368Z "base_url":"http://solr-node2:8984/solr",
2026-05-07T15:34:41.191784426Z "collection":"test-collection",
2026-05-07T15:34:41.191786398Z "shard":"shard1",
2026-05-07T15:34:41.191788682Z "state":"down",
2026-05-07T15:34:41.191790892Z "type":"NRT",
2026-05-07T15:34:41.191792747Z "operation":"ADDREPLICA",
2026-05-07T15:34:41.191794679Z "waitForFinalState":"false"}
2026-05-07T15:34:41.301955403Z 2026-05-07 15:34:41.301 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:41.420069096Z 2026-05-07 15:34:41.419 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-07T15:34:41.583880776Z 2026-05-07 15:34:41.583 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-07T15:34:41.682133349Z 2026-05-07 15:34:41.681 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-07T15:34:42.063310301Z 2026-05-07 15:34:42.062 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-07T15:34:42.063342451Z 2026-05-07 15:34:42.062 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-07T15:34:42.094145357Z 2026-05-07 15:34:42.093 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-07T15:34:43.279654807Z 2026-05-07 15:34:43.279 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-07T15:34:43.306611341Z 2026-05-07 15:34:43.305 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-07T15:34:43.309637045Z 2026-05-07 15:34:43.309 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-07T15:34:43.311063094Z 2026-05-07 15:34:43.310 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-07T15:34:43.311118824Z 2026-05-07 15:34:43.310 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-07T15:34:43.371527307Z 2026-05-07 15:34:43.370 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-07T15:34:43.373892043Z 2026-05-07 15:34:43.373 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-07T15:34:43.375720541Z 2026-05-07 15:34:43.375 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-07T15:34:43.382502051Z 2026-05-07 15:34:43.382 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-07T15:34:43.394839821Z 2026-05-07 15:34:43.394 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-07T15:34:43.402450776Z 2026-05-07 15:34:43.402 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-07T15:34:43.408271561Z 2026-05-07 15:34:43.408 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-07T15:34:43.408362253Z 2026-05-07 15:34:43.408 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-07T15:34:43.416563131Z 2026-05-07 15:34:43.416 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-07T15:34:43.421146685Z 2026-05-07 15:34:43.420 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-07T15:34:43.422282644Z 2026-05-07 15:34:43.422 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-07T15:34:43.433409592Z 2026-05-07 15:34:43.433 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-07T15:34:43.433422993Z 2026-05-07 15:34:43.433 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-07T15:34:43.433492750Z 2026-05-07 15:34:43.433 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-07T15:34:43.434464623Z 2026-05-07 15:34:43.434 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-07T15:34:43.434899128Z 2026-05-07 15:34:43.434 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-07T15:34:43.435730380Z 2026-05-07 15:34:43.435 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72093707922833408-core_node3-n_0000000000
2026-05-07T15:34:43.442036715Z 2026-05-07 15:34:43.441 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-07T15:34:43.548229083Z 2026-05-07 15:34:43.547 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:43.551238190Z 2026-05-07 15:34:43.550 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-07T15:34:43.557411727Z 2026-05-07 15:34:43.557 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2142
2026-05-07T15:34:43.570814527Z 2026-05-07 15:34:43.570 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-07T15:34:43.657384698Z 2026-05-07 15:34:43.656 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:43.659825130Z 2026-05-07 15:34:43.659 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2587
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1604526,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-07T15:34:21.220405061Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-07T15:34:29.547988503Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-07T15:34:29.548013020Z It should be set to 65000 to avoid operational disruption.
2026-05-07T15:34:29.548016112Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-07T15:34:29.548320939Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-07T15:34:29.548326632Z
2026-05-07T15:34:29.557323203Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-07T15:34:29.565006122Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-07T15:34:29.630069622Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-07T15:34:29.664194626Z WARNING: A command line option has enabled the Security Manager
2026-05-07T15:34:29.664211234Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-07T15:34:34.694249454Z 2026-05-07 15:34:34.691 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-07T15:34:34.952894795Z 2026-05-07 15:34:34.952 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-07T15:34:34.978320809Z 2026-05-07 15:34:34.977 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-07T15:34:34.985059950Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-07T15:34:34.985152100Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-07T15:34:34.985283011Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-07T15:34:34.985416600Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-07T15:34:34.984771319Z
2026-05-07T15:34:34.987918959Z 2026-05-07 15:34:34.987 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-07T15:34:35.009504908Z 2026-05-07 15:34:35.008 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-07T15:34:35.032606663Z 2026-05-07 15:34:35.031 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-07T15:34:35.165927949Z 2026-05-07 15:34:35.165 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-07T15:34:35.181517562Z 2026-05-07 15:34:35.181 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-07T15:34:35.181534294Z 2026-05-07 15:34:35.181 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-07T15:34:35.181619465Z 2026-05-07 15:34:35.181 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-07T15:34:35.290024969Z 2026-05-07 15:34:35.289 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-07T15:34:35.290112847Z 2026-05-07 15:34:35.289 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-07T15:34:35.290230207Z 2026-05-07 15:34:35.289 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-07T15:34:35.323776590Z 2026-05-07 15:34:35.323 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-07T15:34:35.970114196Z 2026-05-07 15:34:35.969 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-07T15:34:36.109429958Z 2026-05-07 15:34:36.108 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-07T15:34:36.111204221Z 2026-05-07 15:34:36.109 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-07T15:34:36.118785183Z 2026-05-07 15:34:36.118 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-07T15:34:36.123328353Z 2026-05-07 15:34:36.122 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-07T15:34:36.126882317Z 2026-05-07 15:34:36.126 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-07T15:34:36.126986192Z 2026-05-07 15:34:36.126 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-07T15:34:36.153995511Z 2026-05-07 15:34:36.153 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-07T15:34:36.161572653Z 2026-05-07 15:34:36.161 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-07T15:34:36.179750583Z 2026-05-07 15:34:36.179 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-07T15:34:36.190586816Z 2026-05-07 15:34:36.190 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-07T15:34:36.200320952Z 2026-05-07 15:34:36.200 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-07T15:34:36.299874944Z 2026-05-07 15:34:36.299 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-07T15:34:36.340887144Z 2026-05-07 15:34:36.340 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-07T15:34:36.619868257Z 2026-05-07 15:34:36.618 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-07T15:34:36.790705921Z 2026-05-07 15:34:36.789 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-07T15:34:37.050641211Z 2026-05-07 15:34:37.050 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-07T15:34:37.177343741Z 2026-05-07 15:34:37.176 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-07T15:34:37.183109916Z 2026-05-07 15:34:37.182 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_07.request.log
2026-05-07T15:34:37.186370044Z 2026-05-07 15:34:37.185 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-07T15:34:37.187039605Z 2026-05-07 15:34:37.186 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @7623ms
2026-05-07T15:34:37.348510890Z 2026-05-07 15:34:37.347 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=46
2026-05-07T15:34:37.406163297Z 2026-05-07 15:34:37.405 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-07T15:34:41.416447933Z 2026-05-07 15:34:41.415 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-07T15:34:41.473217091Z 2026-05-07 15:34:41.472 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-07T15:34:41.533769025Z 2026-05-07 15:34:41.533 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-07T15:34:41.831365838Z 2026-05-07 15:34:41.830 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-07T15:34:41.831391287Z 2026-05-07 15:34:41.830 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-07T15:34:41.851173829Z 2026-05-07 15:34:41.850 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-07T15:34:42.314735273Z 2026-05-07 15:34:42.314 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-07T15:34:42.337211868Z 2026-05-07 15:34:42.336 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-07T15:34:42.344920717Z 2026-05-07 15:34:42.344 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-07T15:34:42.346208235Z 2026-05-07 15:34:42.345 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-07T15:34:42.346221350Z 2026-05-07 15:34:42.345 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-07T15:34:42.393008493Z 2026-05-07 15:34:42.392 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-07T15:34:42.395969705Z 2026-05-07 15:34:42.395 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-07T15:34:42.397840006Z 2026-05-07 15:34:42.397 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-07T15:34:42.399456887Z 2026-05-07 15:34:42.399 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-07T15:34:42.414707900Z 2026-05-07 15:34:42.414 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-07T15:34:42.422262926Z 2026-05-07 15:34:42.421 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-07T15:34:42.427841127Z 2026-05-07 15:34:42.427 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-07T15:34:42.427948587Z 2026-05-07 15:34:42.427 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-07T15:34:42.435544609Z 2026-05-07 15:34:42.435 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-07T15:34:42.443986820Z 2026-05-07 15:34:42.443 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-07T15:34:42.445606038Z 2026-05-07 15:34:42.445 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-07T15:34:42.462153833Z 2026-05-07 15:34:42.461 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-07T15:34:42.462171455Z 2026-05-07 15:34:42.461 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-07T15:34:42.462174395Z 2026-05-07 15:34:42.461 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-07T15:34:42.463441171Z 2026-05-07 15:34:42.463 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-07T15:34:42.465130922Z 2026-05-07 15:34:42.463 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-07T15:34:42.465150820Z 2026-05-07 15:34:42.464 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72093707922833412-core_node6-n_0000000000
2026-05-07T15:34:42.478622293Z 2026-05-07 15:34:42.478 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-07T15:34:42.589422309Z 2026-05-07 15:34:42.588 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:42.593704173Z 2026-05-07 15:34:42.593 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-07T15:34:42.603637594Z 2026-05-07 15:34:42.603 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1189
2026-05-07T15:34:42.770598780Z 2026-05-07 15:34:42.770 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:43.548385139Z 2026-05-07 15:34:43.547 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:43.657730984Z 2026-05-07 15:34:43.657 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1604934,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-07T15:34:29.196509578Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-07T15:34:29.610489764Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-07T15:34:29.610532413Z It should be set to 65000 to avoid operational disruption.
2026-05-07T15:34:29.610536716Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-07T15:34:29.610883268Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-07T15:34:29.610895101Z
2026-05-07T15:34:29.619746480Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-07T15:34:29.627425497Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-07T15:34:29.668373419Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-07T15:34:29.698487806Z WARNING: A command line option has enabled the Security Manager
2026-05-07T15:34:29.698506095Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-07T15:34:30.885204692Z 2026-05-07 15:34:30.882 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-07T15:34:31.157728580Z 2026-05-07 15:34:31.157 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-07T15:34:31.184957477Z 2026-05-07 15:34:31.184 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-07T15:34:31.192040229Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-07T15:34:31.192158366Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-07T15:34:31.192274265Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-07T15:34:31.192457352Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-07T15:34:31.191868775Z
2026-05-07T15:34:31.194919642Z 2026-05-07 15:34:31.194 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-07T15:34:31.217067104Z 2026-05-07 15:34:31.216 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-07T15:34:31.245535611Z 2026-05-07 15:34:31.245 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-07T15:34:31.382569662Z 2026-05-07 15:34:31.381 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-07T15:34:31.403012176Z 2026-05-07 15:34:31.402 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-07T15:34:31.403084638Z 2026-05-07 15:34:31.402 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-07T15:34:31.403208257Z 2026-05-07 15:34:31.402 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-07T15:34:31.523370387Z 2026-05-07 15:34:31.522 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-07T15:34:31.523499263Z 2026-05-07 15:34:31.523 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-07T15:34:31.523589646Z 2026-05-07 15:34:31.523 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-07T15:34:31.562453878Z 2026-05-07 15:34:31.561 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-07T15:34:32.312555004Z 2026-05-07 15:34:32.311 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-07T15:34:32.456948546Z 2026-05-07 15:34:32.456 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-07T15:34:32.458146432Z 2026-05-07 15:34:32.456 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-07T15:34:32.465636800Z 2026-05-07 15:34:32.465 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-07T15:34:32.470065252Z 2026-05-07 15:34:32.469 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-07T15:34:32.473400781Z 2026-05-07 15:34:32.473 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-07T15:34:32.473544795Z 2026-05-07 15:34:32.473 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-07T15:34:32.500461243Z 2026-05-07 15:34:32.499 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-07T15:34:32.507621331Z 2026-05-07 15:34:32.507 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-07T15:34:32.526375462Z 2026-05-07 15:34:32.525 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-07T15:34:32.536011184Z 2026-05-07 15:34:32.535 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-07T15:34:32.544458799Z 2026-05-07 15:34:32.544 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-07T15:34:32.554730250Z 2026-05-07 15:34:32.554 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-07T15:34:32.593841437Z 2026-05-07 15:34:32.593 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-07T15:34:32.861458454Z 2026-05-07 15:34:32.860 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-07T15:34:33.034794603Z 2026-05-07 15:34:33.034 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-07T15:34:33.291012430Z 2026-05-07 15:34:33.290 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-07T15:34:33.435019440Z 2026-05-07 15:34:33.434 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-07T15:34:33.440365212Z 2026-05-07 15:34:33.439 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_07.request.log
2026-05-07T15:34:33.443480642Z 2026-05-07 15:34:33.443 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-07T15:34:33.444253144Z 2026-05-07 15:34:33.444 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3818ms
2026-05-07T15:34:36.208932394Z 2026-05-07 15:34:36.208 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-07T15:34:39.248710859Z 2026-05-07 15:34:39.244 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=77
2026-05-07T15:34:39.317798087Z 2026-05-07 15:34:39.317 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16
2026-05-07T15:34:41.416479195Z 2026-05-07 15:34:41.415 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-07T15:34:41.491868979Z 2026-05-07 15:34:41.491 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-07T15:34:41.557701473Z 2026-05-07 15:34:41.557 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-07T15:34:41.878394054Z 2026-05-07 15:34:41.877 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-07T15:34:41.878417657Z 2026-05-07 15:34:41.878 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-07T15:34:41.896881775Z 2026-05-07 15:34:41.896 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-07T15:34:42.318457834Z 2026-05-07 15:34:42.317 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-07T15:34:42.340397487Z 2026-05-07 15:34:42.339 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-07T15:34:42.345830046Z 2026-05-07 15:34:42.345 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-07T15:34:42.346854540Z 2026-05-07 15:34:42.346 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-07T15:34:42.346902199Z 2026-05-07 15:34:42.346 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-07T15:34:42.393155336Z 2026-05-07 15:34:42.392 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-07T15:34:42.396135835Z 2026-05-07 15:34:42.395 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-07T15:34:42.399573394Z 2026-05-07 15:34:42.397 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-07T15:34:42.400913245Z 2026-05-07 15:34:42.400 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-07T15:34:42.416228284Z 2026-05-07 15:34:42.415 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-07T15:34:42.424601836Z 2026-05-07 15:34:42.424 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-07T15:34:42.431400137Z 2026-05-07 15:34:42.430 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-07T15:34:42.431423238Z 2026-05-07 15:34:42.430 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-07T15:34:42.437108790Z 2026-05-07 15:34:42.436 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-07T15:34:42.445737216Z 2026-05-07 15:34:42.445 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-07T15:34:42.447060304Z 2026-05-07 15:34:42.446 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-07T15:34:42.462535580Z 2026-05-07 15:34:42.462 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-07T15:34:42.462557198Z 2026-05-07 15:34:42.462 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-07T15:34:42.462607826Z 2026-05-07 15:34:42.462 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-07T15:34:42.464041554Z 2026-05-07 15:34:42.463 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-07T15:34:42.464646535Z 2026-05-07 15:34:42.464 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-07T15:34:42.465820094Z 2026-05-07 15:34:42.465 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72093707922833410-core_node5-n_0000000000
2026-05-07T15:34:42.478926348Z 2026-05-07 15:34:42.478 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-07T15:34:42.589219864Z 2026-05-07 15:34:42.588 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:42.593086471Z 2026-05-07 15:34:42.592 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-07T15:34:42.601884490Z 2026-05-07 15:34:42.601 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1187
2026-05-07T15:34:42.770631488Z 2026-05-07 15:34:42.770 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:43.548697968Z 2026-05-07 15:34:43.548 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-07T15:34:43.657589629Z 2026-05-07 15:34:43.657 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1605064,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-07T15:34:29.406068335Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 29440 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# f71bdaed5253: Pulling fs layer
# 4b0d3fb3d62d: Pulling fs layer
# 651268e22bd5: Pulling fs layer
# ed01378be80d: Pulling fs layer
# df4326c4e3ea: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# ed01378be80d: Waiting
# df4326c4e3ea: Waiting
# 4f4fb700ef54: Waiting
# 651268e22bd5: Verifying Checksum
# 651268e22bd5: Download complete
# 4b0d3fb3d62d: Verifying Checksum
# 4b0d3fb3d62d: Download complete
# ed01378be80d: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# df4326c4e3ea: Verifying Checksum
# df4326c4e3ea: Download complete
# f71bdaed5253: Verifying Checksum
# f71bdaed5253: Download complete
# f71bdaed5253: Pull complete
# 4b0d3fb3d62d: Pull complete
# 651268e22bd5: Pull complete
# ed01378be80d: Pull complete
# df4326c4e3ea: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:c596878100cb3e6f257b2bf7137884cf5710f73b2d05f24e05be9fa69347feb9
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# b25f78e2ffe088e2d6243e61a0705ea92b2690e6856ecaaef2ef814c2b7af237
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# e6a5b5569c2a6e07a0d65af2c20e4ba09b09ca39a32d7a61b6850976395ac542
# Starting solr-node2 connected to first node's ZooKeeper
# d1b98ffdc547f4c840f7949f3226dbb05f4f181fdbefd9f63ff802c0d543457c
# Started solr-node3 connected to first node's ZooKeeper
# 1eca627bc7e335429cf814f04317c41effd6bed666bb3a26b16cc2b639548cf4
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-07 15:34:40.825; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-07 15:34:40.860; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14900 ms
ok 85 snapshot lifecycle # in 4591 ms
ok 86 snapshot list # in 3208 ms
ok 87 snapshot describe # in 3222 ms
ok 88 start solr with ssl # in 15356 ms
ok 89 use different hostname when not checking peer-name # in 23758 ms
ok 90 start solr with ssl and auth # in 14505 ms
ok 91 start solr with client truststore and security manager # in 15921 ms
ok 92 start solr with mTLS needed # in 44841 ms
ok 93 start solr with mTLS wanted # in 34315 ms
ok 94 test keystore reload # in 56779 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15482 ms
ok 96 stop command for single port # in 17848 ms
ok 97 check stop command doesn't hang # in 207 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7974 ms
ok 99 deprecated system properties converted to modern properties # in 195275 ms
ok 100 start with custom jetty options # in 7804 ms
ok 101 bootstrapping a configset # in 8862 ms
ok 102 status detects locally running solr # in 9151 ms
ok 103 status with --solr-url from user # in 7757 ms
ok 104 status with --port from user # in 7849 ms
ok 105 multiple connection options are prevented # in 819 ms
ok 106 status with invalid --solr-url from user # in 1258 ms
ok 107 status with --short format # in 196020 ms
ok 108 searching solr via locally executed streaming expression # in 1612 ms
ok 109 searching solr via remotely executed streaming expression # in 1193 ms
ok 110 variable interpolation # in 1584 ms
ok 111 searching solr without credentials fails with error # in 1057 ms
ok 112 --version returns Solr version # in 683 ms
ok 113 version as direct tool call still runs # in 703 ms
ok 114 short help # in 742 ms
ok 115 short help is inferred # in 722 ms
ok 116 long help # in 746 ms
ok 117 running subcommands with zk is prevented # in 690 ms
ok 118 listing out files # in 2087 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7534 ms
ok 120 copying files around # in 10586 ms
ok 121 upconfig # in 2469 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1059 ms
ok 123 downconfig # in 1124 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1362 ms
ok 125 env var ZK_HOST is honored # in 3075 ms
ok 126 Cleanup # in 1804 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 42s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/lm5yz75eduq52
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 201 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/201/
No tests ran.
Build Log:
[...truncated 418 lines...]
# ERROR: Server refused connection at: http://localhost:40507/solr/test_args/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:40507/solr/test_args/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #10 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/79-10
# Last output:
# POSTing args to http://localhost:40507/solr/test_args/update...
# PostTool: WARNING: Connection error (is Solr running at http://localhost:40507/solr/test_args/update ?): java.net.ConnectException: Connection refused
# PostTool: WARNING: IOException while posting data: java.net.ConnectException: Connection refused
# PostTool: WARNING: IOException while reading response: java.net.ConnectException: Connection refused
# COMMITting Solr index changes to http://localhost:40507/solr/test_args/update...
#
# ERROR: Server refused connection at: http://localhost:40507/solr/test_args/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:40507/solr/test_args/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 80 stdin mode # in 1043 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 201)
# `assert_output --partial "Created collection 'test_stdin'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'test_stdin'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40507.
#
# ERROR: Server refused connection at: http://localhost:40507/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40507.
#
# ERROR: Server refused connection at: http://localhost:40507/solr/admin/info/system
not ok 81 verbose echo the Solr response # in 1053 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40507.
#
# ERROR: Server refused connection at: http://localhost:40507/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40507.
#
# ERROR: Server refused connection at: http://localhost:40507/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8520 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 8 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-06T12:35:49.316520926Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-06T12:35:49.316573662Z It should be set to 65000 to avoid operational disruption.
2026-05-06T12:35:49.316577422Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-06T12:35:49.317080320Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-06T12:35:49.317088348Z
2026-05-06T12:35:49.329246767Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-06T12:35:49.338330197Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-06T12:35:49.386165780Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-06T12:35:49.426790862Z WARNING: A command line option has enabled the Security Manager
2026-05-06T12:35:49.426823190Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-06T12:35:50.648038930Z 2026-05-06 12:35:50.644 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-06T12:35:50.940835238Z 2026-05-06 12:35:50.940 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-06T12:35:50.972981113Z 2026-05-06 12:35:50.972 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-06T12:35:50.982358783Z 2026-05-06 12:35:50.981 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-06T12:35:50.982394569Z 2026-05-06 12:35:50.981 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-06T12:35:50.982454408Z 2026-05-06 12:35:50.981 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-06T12:35:50.982611394Z 2026-05-06 12:35:50.981 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-06T12:35:50.981802986Z
2026-05-06T12:35:50.986277153Z 2026-05-06 12:35:50.985 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-06T12:35:51.016631010Z 2026-05-06 12:35:51.015 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-06T12:35:51.035797926Z 2026-05-06 12:35:51.034 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-06T12:35:51.035830756Z 2026-05-06 12:35:51.035 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-06T12:35:51.080201236Z 2026-05-06 12:35:51.079 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-06T12:35:51.978382960Z 2026-05-06 12:35:51.977 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-06T12:35:52.264199326Z 2026-05-06 12:35:52.262 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-06T12:35:52.270436695Z 2026-05-06 12:35:52.270 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-06T12:35:52.271342845Z 2026-05-06 12:35:52.270 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-06T12:35:52.274176778Z 2026-05-06 12:35:52.273 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-06T12:35:52.274248300Z 2026-05-06 12:35:52.273 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-06T12:35:52.317114756Z 2026-05-06 12:35:52.316 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-06T12:35:52.778276176Z 2026-05-06 12:35:52.777 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-06T12:35:52.790644616Z 2026-05-06 12:35:52.790 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-06T12:35:52.840081859Z 2026-05-06 12:35:52.839 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-06T12:35:52.881185804Z 2026-05-06 12:35:52.880 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-06T12:35:52.881253796Z 2026-05-06 12:35:52.880 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-06T12:35:53.006699899Z 2026-05-06 12:35:53.005 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-06T12:35:53.028307324Z 2026-05-06 12:35:53.027 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-06T12:35:53.042573605Z 2026-05-06 12:35:53.042 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-06T12:35:53.048536630Z 2026-05-06 12:35:53.048 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72087343479717888-solr-node1:8983_solr-n_0000000000) starting
2026-05-06T12:35:53.109613322Z 2026-05-06 12:35:53.109 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-06T12:35:53.110582909Z 2026-05-06 12:35:53.110 INFO (OverseerStateUpdate-72087343479717888-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-06T12:35:53.119227268Z 2026-05-06 12:35:53.118 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-06T12:35:53.160616543Z 2026-05-06 12:35:53.160 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-06T12:35:53.540726958Z 2026-05-06 12:35:53.540 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-06T12:35:53.710361236Z 2026-05-06 12:35:53.709 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-06T12:35:53.961345586Z 2026-05-06 12:35:53.960 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-06T12:35:54.092208127Z 2026-05-06 12:35:54.091 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-06T12:35:54.098449300Z 2026-05-06 12:35:54.097 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_06.request.log
2026-05-06T12:35:54.105322545Z 2026-05-06 12:35:54.104 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-06T12:35:54.106815907Z 2026-05-06 12:35:54.106 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4769ms
2026-05-06T12:35:54.309684702Z 2026-05-06 12:35:54.309 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=40
2026-05-06T12:35:54.369960656Z 2026-05-06 12:35:54.369 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-05-06T12:35:57.466336705Z 2026-05-06 12:35:57.464 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-06T12:35:57.770769616Z 2026-05-06 12:35:57.770 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-06T12:36:01.035783862Z 2026-05-06 12:36:01.035 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-05-06T12:36:01.062839534Z 2026-05-06 12:36:01.062 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-05-06T12:36:01.086441556Z 2026-05-06 12:36:01.085 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-06T12:36:01.274859830Z 2026-05-06 12:36:01.274 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=5
2026-05-06T12:36:01.298377604Z 2026-05-06 12:36:01.297 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-06T12:36:01.313618142Z 2026-05-06 12:36:01.313 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-06T12:36:01.321625936Z 2026-05-06 12:36:01.321 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:01.355726376Z 2026-05-06 12:36:01.355 INFO (OverseerStateUpdate-72087343479717888-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-06T12:36:01.355744858Z "core":"test-collection_shard3_replica_n1",
2026-05-06T12:36:01.355747620Z "node_name":"solr-node1:8983_solr",
2026-05-06T12:36:01.355750219Z "base_url":"http://solr-node1:8983/solr",
2026-05-06T12:36:01.355752355Z "collection":"test-collection",
2026-05-06T12:36:01.355754402Z "shard":"shard3",
2026-05-06T12:36:01.355756364Z "state":"down",
2026-05-06T12:36:01.355758324Z "type":"NRT",
2026-05-06T12:36:01.355760872Z "operation":"ADDREPLICA",
2026-05-06T12:36:01.355763335Z "waitForFinalState":"false"}
2026-05-06T12:36:01.363657059Z 2026-05-06 12:36:01.363 INFO (OverseerStateUpdate-72087343479717888-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-06T12:36:01.363671967Z "core":"test-collection_shard2_replica_n2",
2026-05-06T12:36:01.363675278Z "node_name":"solr-node3:8985_solr",
2026-05-06T12:36:01.363677476Z "base_url":"http://solr-node3:8985/solr",
2026-05-06T12:36:01.363679542Z "collection":"test-collection",
2026-05-06T12:36:01.363681628Z "shard":"shard2",
2026-05-06T12:36:01.363683592Z "state":"down",
2026-05-06T12:36:01.363685601Z "type":"NRT",
2026-05-06T12:36:01.363687724Z "operation":"ADDREPLICA",
2026-05-06T12:36:01.363689790Z "waitForFinalState":"false"}
2026-05-06T12:36:01.368316469Z 2026-05-06 12:36:01.367 INFO (OverseerStateUpdate-72087343479717888-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-06T12:36:01.368336725Z "core":"test-collection_shard1_replica_n4",
2026-05-06T12:36:01.368339426Z "node_name":"solr-node2:8984_solr",
2026-05-06T12:36:01.368341576Z "base_url":"http://solr-node2:8984/solr",
2026-05-06T12:36:01.368343629Z "collection":"test-collection",
2026-05-06T12:36:01.368345581Z "shard":"shard1",
2026-05-06T12:36:01.368347500Z "state":"down",
2026-05-06T12:36:01.368349470Z "type":"NRT",
2026-05-06T12:36:01.368351457Z "operation":"ADDREPLICA",
2026-05-06T12:36:01.368353395Z "waitForFinalState":"false"}
2026-05-06T12:36:01.526887661Z 2026-05-06 12:36:01.526 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:01.599052362Z 2026-05-06 12:36:01.598 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-06T12:36:01.686730231Z 2026-05-06 12:36:01.686 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-06T12:36:01.737347483Z 2026-05-06 12:36:01.736 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-06T12:36:02.058344713Z 2026-05-06 12:36:02.055 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-06T12:36:02.058423512Z 2026-05-06 12:36:02.056 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-06T12:36:02.079832489Z 2026-05-06 12:36:02.079 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-06T12:36:02.912159304Z 2026-05-06 12:36:02.910 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-06T12:36:02.976046727Z 2026-05-06 12:36:02.975 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-06T12:36:02.987410628Z 2026-05-06 12:36:02.987 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-06T12:36:02.988845040Z 2026-05-06 12:36:02.988 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-06T12:36:02.988899207Z 2026-05-06 12:36:02.988 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-06T12:36:03.108285396Z 2026-05-06 12:36:03.107 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-06T12:36:03.110741792Z 2026-05-06 12:36:03.110 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-06T12:36:03.113835452Z 2026-05-06 12:36:03.113 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-06T12:36:03.115438614Z 2026-05-06 12:36:03.115 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-06T12:36:03.128333086Z 2026-05-06 12:36:03.127 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-06T12:36:03.138898055Z 2026-05-06 12:36:03.138 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-06T12:36:03.149509856Z 2026-05-06 12:36:03.149 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-06T12:36:03.149537700Z 2026-05-06 12:36:03.149 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-06T12:36:03.162759641Z 2026-05-06 12:36:03.162 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-06T12:36:03.164018914Z 2026-05-06 12:36:03.163 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-06T12:36:03.171701137Z 2026-05-06 12:36:03.171 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-06T12:36:03.180231980Z 2026-05-06 12:36:03.179 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-06T12:36:03.180245363Z 2026-05-06 12:36:03.179 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-06T12:36:03.180249684Z 2026-05-06 12:36:03.179 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-06T12:36:03.181621799Z 2026-05-06 12:36:03.181 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-06T12:36:03.182275384Z 2026-05-06 12:36:03.182 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-06T12:36:03.184610242Z 2026-05-06 12:36:03.184 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72087343479717888-core_node3-n_0000000000
2026-05-06T12:36:03.205514323Z 2026-05-06 12:36:03.205 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-06T12:36:03.310898958Z 2026-05-06 12:36:03.310 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 240] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:03.316204947Z 2026-05-06 12:36:03.315 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-06T12:36:03.324699172Z 2026-05-06 12:36:03.324 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1727
2026-05-06T12:36:03.339265527Z 2026-05-06 12:36:03.337 INFO (qtp985324122-28-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-06T12:36:03.428236052Z 2026-05-06 12:36:03.427 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 246] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:03.431817658Z 2026-05-06 12:36:03.431 INFO (qtp985324122-28-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2147
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1577018,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-06T12:35:49.060861975Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-06T12:35:54.678045104Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-06T12:35:54.678066208Z It should be set to 65000 to avoid operational disruption.
2026-05-06T12:35:54.678069194Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-06T12:35:54.678385282Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-06T12:35:54.678390950Z
2026-05-06T12:35:54.687043318Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-06T12:35:54.694376631Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-06T12:35:54.737879504Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-06T12:35:54.770008991Z WARNING: A command line option has enabled the Security Manager
2026-05-06T12:35:54.770026306Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-06T12:35:55.828282323Z 2026-05-06 12:35:55.825 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-06T12:35:56.097037098Z 2026-05-06 12:35:56.096 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-06T12:35:56.123408958Z 2026-05-06 12:35:56.122 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-06T12:35:56.130732373Z 2026-05-06 12:35:56.130 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-06T12:35:56.131897617Z 2026-05-06 12:35:56.130 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-06T12:35:56.131910189Z 2026-05-06 12:35:56.130 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-06T12:35:56.131913382Z 2026-05-06 12:35:56.130 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-06T12:35:56.130480070Z
2026-05-06T12:35:56.134014001Z 2026-05-06 12:35:56.133 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-06T12:35:56.157865551Z 2026-05-06 12:35:56.157 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-06T12:35:56.183036308Z 2026-05-06 12:35:56.182 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-06T12:35:56.322717208Z 2026-05-06 12:35:56.322 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-06T12:35:56.340161961Z 2026-05-06 12:35:56.339 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-06T12:35:56.340180401Z 2026-05-06 12:35:56.339 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-06T12:35:56.340182973Z 2026-05-06 12:35:56.339 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-06T12:35:56.451322971Z 2026-05-06 12:35:56.450 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-06T12:35:56.451341325Z 2026-05-06 12:35:56.450 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-06T12:35:56.451344211Z 2026-05-06 12:35:56.450 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-06T12:35:56.483716462Z 2026-05-06 12:35:56.483 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-06T12:35:57.188992063Z 2026-05-06 12:35:57.188 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-06T12:35:57.354438310Z 2026-05-06 12:35:57.353 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-06T12:35:57.355720210Z 2026-05-06 12:35:57.353 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-06T12:35:57.367025912Z 2026-05-06 12:35:57.366 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-06T12:35:57.372471308Z 2026-05-06 12:35:57.371 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-06T12:35:57.375971126Z 2026-05-06 12:35:57.375 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-06T12:35:57.376064419Z 2026-05-06 12:35:57.375 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-06T12:35:57.404869697Z 2026-05-06 12:35:57.404 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-06T12:35:57.413268219Z 2026-05-06 12:35:57.412 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-06T12:35:57.432211809Z 2026-05-06 12:35:57.431 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-06T12:35:57.442516625Z 2026-05-06 12:35:57.442 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-06T12:35:57.455128782Z 2026-05-06 12:35:57.454 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-06T12:35:57.466137800Z 2026-05-06 12:35:57.465 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-06T12:35:57.554772200Z 2026-05-06 12:35:57.553 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-06T12:35:57.771456454Z 2026-05-06 12:35:57.770 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-06T12:35:57.849063350Z 2026-05-06 12:35:57.848 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-06T12:35:58.013323210Z 2026-05-06 12:35:58.012 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-06T12:35:58.265590547Z 2026-05-06 12:35:58.265 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-06T12:35:58.393211835Z 2026-05-06 12:35:58.392 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-06T12:35:58.398661080Z 2026-05-06 12:35:58.398 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_06.request.log
2026-05-06T12:35:58.403043251Z 2026-05-06 12:35:58.402 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-06T12:35:58.405604998Z 2026-05-06 12:35:58.405 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3712ms
2026-05-06T12:35:58.555542591Z 2026-05-06 12:35:58.554 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=43
2026-05-06T12:35:58.605473703Z 2026-05-06 12:35:58.604 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-06T12:36:01.598938080Z 2026-05-06 12:36:01.598 INFO (qtp985324122-59-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-06T12:36:01.645858256Z 2026-05-06 12:36:01.645 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-06T12:36:01.703682805Z 2026-05-06 12:36:01.703 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-06T12:36:02.002746763Z 2026-05-06 12:36:02.002 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-06T12:36:02.002778101Z 2026-05-06 12:36:02.002 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-06T12:36:02.021419500Z 2026-05-06 12:36:02.020 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-06T12:36:02.429481199Z 2026-05-06 12:36:02.429 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-06T12:36:02.447157479Z 2026-05-06 12:36:02.446 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-06T12:36:02.450540232Z 2026-05-06 12:36:02.450 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-06T12:36:02.451400639Z 2026-05-06 12:36:02.450 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-06T12:36:02.451438332Z 2026-05-06 12:36:02.451 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-06T12:36:02.489734818Z 2026-05-06 12:36:02.489 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-06T12:36:02.492540394Z 2026-05-06 12:36:02.492 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-06T12:36:02.494363593Z 2026-05-06 12:36:02.494 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-06T12:36:02.496048004Z 2026-05-06 12:36:02.495 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-06T12:36:02.507110586Z 2026-05-06 12:36:02.506 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-06T12:36:02.514346942Z 2026-05-06 12:36:02.513 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-06T12:36:02.523426324Z 2026-05-06 12:36:02.523 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-06T12:36:02.523452947Z 2026-05-06 12:36:02.523 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-06T12:36:02.529430671Z 2026-05-06 12:36:02.529 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-06T12:36:02.547032894Z 2026-05-06 12:36:02.546 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-06T12:36:02.548391243Z 2026-05-06 12:36:02.548 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-06T12:36:02.565557160Z 2026-05-06 12:36:02.565 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-06T12:36:02.565587055Z 2026-05-06 12:36:02.565 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-06T12:36:02.565686619Z 2026-05-06 12:36:02.565 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-06T12:36:02.566991684Z 2026-05-06 12:36:02.566 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-06T12:36:02.567578325Z 2026-05-06 12:36:02.567 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-06T12:36:02.568651023Z 2026-05-06 12:36:02.568 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72087343479717891-core_node6-n_0000000000
2026-05-06T12:36:02.583286898Z 2026-05-06 12:36:02.582 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-06T12:36:02.727635840Z 2026-05-06 12:36:02.727 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:02.732387657Z 2026-05-06 12:36:02.732 INFO (qtp985324122-59-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-06T12:36:02.742573530Z 2026-05-06 12:36:02.742 INFO (qtp985324122-59-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1145
2026-05-06T12:36:02.913475729Z 2026-05-06 12:36:02.913 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:03.031434125Z 2026-05-06 12:36:03.030 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 225] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:03.310989678Z 2026-05-06 12:36:03.310 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 240] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:03.428960817Z 2026-05-06 12:36:03.428 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 246] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1577433,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-06T12:35:54.492825072Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-06T12:35:54.881128799Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-06T12:35:54.881169609Z It should be set to 65000 to avoid operational disruption.
2026-05-06T12:35:54.881172640Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-06T12:35:54.881414867Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-06T12:35:54.881418598Z
2026-05-06T12:35:54.890060486Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-06T12:35:54.897418354Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-06T12:35:54.943116944Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-06T12:35:54.974876731Z WARNING: A command line option has enabled the Security Manager
2026-05-06T12:35:54.974905554Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-06T12:35:55.975165587Z 2026-05-06 12:35:55.971 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-06T12:35:56.234125567Z 2026-05-06 12:35:56.233 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-06T12:35:56.258555440Z 2026-05-06 12:35:56.258 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-06T12:35:56.265457101Z 2026-05-06 12:35:56.264 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-06T12:35:56.265547965Z 2026-05-06 12:35:56.265 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-06T12:35:56.265642038Z 2026-05-06 12:35:56.265 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-06T12:35:56.265749037Z 2026-05-06 12:35:56.265 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-06T12:35:56.265248719Z
2026-05-06T12:35:56.268391627Z 2026-05-06 12:35:56.267 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-06T12:35:56.289437565Z 2026-05-06 12:35:56.288 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-06T12:35:56.312083640Z 2026-05-06 12:35:56.311 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-06T12:35:56.443288282Z 2026-05-06 12:35:56.442 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-06T12:35:56.461380180Z 2026-05-06 12:35:56.460 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-06T12:35:56.461458185Z 2026-05-06 12:35:56.461 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-06T12:35:56.461553401Z 2026-05-06 12:35:56.461 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-06T12:35:56.569693497Z 2026-05-06 12:35:56.569 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-06T12:35:56.569823971Z 2026-05-06 12:35:56.569 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-06T12:35:56.569921149Z 2026-05-06 12:35:56.569 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-06T12:35:56.602447281Z 2026-05-06 12:35:56.601 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-06T12:35:57.533582233Z 2026-05-06 12:35:57.533 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-06T12:35:57.664768915Z 2026-05-06 12:35:57.664 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-06T12:35:57.666063303Z 2026-05-06 12:35:57.664 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-06T12:35:57.678589302Z 2026-05-06 12:35:57.678 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-06T12:35:57.683485681Z 2026-05-06 12:35:57.683 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-06T12:35:57.687882802Z 2026-05-06 12:35:57.687 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-06T12:35:57.687969886Z 2026-05-06 12:35:57.687 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-06T12:35:57.717360752Z 2026-05-06 12:35:57.716 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-06T12:35:57.724636175Z 2026-05-06 12:35:57.724 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-06T12:35:57.744423067Z 2026-05-06 12:35:57.743 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-06T12:35:57.754505581Z 2026-05-06 12:35:57.754 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-06T12:35:57.763525447Z 2026-05-06 12:35:57.763 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-06T12:35:57.772159348Z 2026-05-06 12:35:57.771 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-06T12:35:57.810267275Z 2026-05-06 12:35:57.809 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-06T12:35:58.053549815Z 2026-05-06 12:35:58.053 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-06T12:35:58.210565128Z 2026-05-06 12:35:58.210 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-06T12:35:58.448772231Z 2026-05-06 12:35:58.448 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-06T12:35:58.566139995Z 2026-05-06 12:35:58.565 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-06T12:35:58.571723556Z 2026-05-06 12:35:58.571 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_06.request.log
2026-05-06T12:35:58.574808582Z 2026-05-06 12:35:58.574 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-06T12:35:58.575649059Z 2026-05-06 12:35:58.575 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3679ms
2026-05-06T12:35:59.943493698Z 2026-05-06 12:35:59.937 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=62
2026-05-06T12:35:59.997575969Z 2026-05-06 12:35:59.996 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-05-06T12:36:01.601594683Z 2026-05-06 12:36:01.600 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-06T12:36:01.664775421Z 2026-05-06 12:36:01.664 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-06T12:36:01.855456545Z 2026-05-06 12:36:01.854 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-05-06T12:36:02.143973989Z 2026-05-06 12:36:02.143 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-06T12:36:02.143995937Z 2026-05-06 12:36:02.143 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-06T12:36:02.166212310Z 2026-05-06 12:36:02.165 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-06T12:36:02.601172939Z 2026-05-06 12:36:02.600 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-06T12:36:02.623134043Z 2026-05-06 12:36:02.622 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-06T12:36:02.630906110Z 2026-05-06 12:36:02.630 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-06T12:36:02.632067393Z 2026-05-06 12:36:02.631 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-06T12:36:02.632079572Z 2026-05-06 12:36:02.631 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-06T12:36:02.727225890Z 2026-05-06 12:36:02.726 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-06T12:36:02.730379697Z 2026-05-06 12:36:02.729 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-06T12:36:02.732167706Z 2026-05-06 12:36:02.731 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-06T12:36:02.735687041Z 2026-05-06 12:36:02.735 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-06T12:36:02.747749518Z 2026-05-06 12:36:02.747 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-06T12:36:02.755204695Z 2026-05-06 12:36:02.754 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-06T12:36:02.762879146Z 2026-05-06 12:36:02.761 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-06T12:36:02.762901441Z 2026-05-06 12:36:02.761 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-06T12:36:02.767877539Z 2026-05-06 12:36:02.767 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-06T12:36:02.778794118Z 2026-05-06 12:36:02.777 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-06T12:36:02.779588648Z 2026-05-06 12:36:02.779 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-06T12:36:02.795817973Z 2026-05-06 12:36:02.795 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-06T12:36:02.795909300Z 2026-05-06 12:36:02.795 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-06T12:36:02.795921093Z 2026-05-06 12:36:02.795 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-06T12:36:02.797544953Z 2026-05-06 12:36:02.797 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-06T12:36:02.798107291Z 2026-05-06 12:36:02.797 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-06T12:36:02.799143770Z 2026-05-06 12:36:02.798 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72087343479717892-core_node5-n_0000000000
2026-05-06T12:36:02.806414177Z 2026-05-06 12:36:02.806 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-06T12:36:02.913699483Z 2026-05-06 12:36:02.913 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:02.918588091Z 2026-05-06 12:36:02.918 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-06T12:36:02.926823571Z 2026-05-06 12:36:02.926 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1329
2026-05-06T12:36:03.031513204Z 2026-05-06 12:36:03.031 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 225] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:03.311079911Z 2026-05-06 12:36:03.310 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 240] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-06T12:36:03.428960550Z 2026-05-06 12:36:03.428 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 246] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1577616,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-06T12:35:54.676461464Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 25467 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 0ed719aef393: Waiting
# 66b76b382631: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 601f2c23751f: Waiting
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 4f4fb700ef54: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# 53c0d5ac4fc8: Pulling fs layer
# 3181eac00d2f: Pulling fs layer
# 7aab65890505: Pulling fs layer
# d17614f6af2c: Pulling fs layer
# fa688e0f8ea3: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# d17614f6af2c: Waiting
# 4f4fb700ef54: Waiting
# fa688e0f8ea3: Waiting
# 3181eac00d2f: Verifying Checksum
# 3181eac00d2f: Download complete
# d17614f6af2c: Verifying Checksum
# d17614f6af2c: Download complete
# 7aab65890505: Verifying Checksum
# 7aab65890505: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# fa688e0f8ea3: Verifying Checksum
# fa688e0f8ea3: Download complete
# 53c0d5ac4fc8: Download complete
# 53c0d5ac4fc8: Pull complete
# 3181eac00d2f: Pull complete
# 7aab65890505: Pull complete
# d17614f6af2c: Pull complete
# fa688e0f8ea3: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:fd2d4da7737cc83c109027cedeb95dc0f2fafb4ca89f5c45aae8be19e0740d25
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 9b508fbd6498b61a2ee8ad7198cfaa37219d4477fec3b1874f064a4652f135a2
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 6f52214fcbebe3157d39d125963f9dc3df9460c031afd2033bdb7263ec258c9e
# Starting solr-node2 connected to first node's ZooKeeper
# 02ab42063e6e12ea7623be4a38e864d7ca17fce2dd2dcd90f0d4ad6de4af889f
# Started solr-node3 connected to first node's ZooKeeper
# c3fb7a77acc955f516009347c725ead22e1a35c0c5cff5bbbd147e72cb3a32f3
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-06 12:36:01.113; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-06 12:36:01.142; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14410 ms
ok 85 snapshot lifecycle # in 4479 ms
ok 86 snapshot list # in 3115 ms
ok 87 snapshot describe # in 3112 ms
ok 88 start solr with ssl # in 15122 ms
ok 89 use different hostname when not checking peer-name # in 23672 ms
ok 90 start solr with ssl and auth # in 14094 ms
ok 91 start solr with client truststore and security manager # in 15637 ms
ok 92 start solr with mTLS needed # in 43819 ms
ok 93 start solr with mTLS wanted # in 32301 ms
ok 94 test keystore reload # in 55959 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15454 ms
ok 96 stop command for single port # in 17574 ms
ok 97 check stop command doesn't hang # in 234 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7716 ms
ok 99 deprecated system properties converted to modern properties # in 195169 ms
ok 100 start with custom jetty options # in 7732 ms
ok 101 bootstrapping a configset # in 8848 ms
ok 102 status detects locally running solr # in 9109 ms
ok 103 status with --solr-url from user # in 7623 ms
ok 104 status with --port from user # in 7856 ms
ok 105 multiple connection options are prevented # in 810 ms
ok 106 status with invalid --solr-url from user # in 1176 ms
ok 107 status with --short format # in 195822 ms
ok 108 searching solr via locally executed streaming expression # in 1643 ms
ok 109 searching solr via remotely executed streaming expression # in 1187 ms
ok 110 variable interpolation # in 1616 ms
ok 111 searching solr without credentials fails with error # in 1082 ms
ok 112 --version returns Solr version # in 732 ms
ok 113 version as direct tool call still runs # in 713 ms
ok 114 short help # in 710 ms
ok 115 short help is inferred # in 723 ms
ok 116 long help # in 741 ms
ok 117 running subcommands with zk is prevented # in 690 ms
ok 118 listing out files # in 2077 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7449 ms
ok 120 copying files around # in 10569 ms
ok 121 upconfig # in 2457 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1050 ms
ok 123 downconfig # in 1111 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1363 ms
ok 125 env var ZK_HOST is honored # in 3044 ms
ok 126 Cleanup # in 1802 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 33s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/iwjwkt7gqiwls
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 200 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/200/
No tests ran.
Build Log:
[...truncated 426 lines...]
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:41435/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:41435.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:41435.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:41435/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c monitors_verbose -s http://localhost:41435 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-05 12:33:02.584; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-05 12:33:02.588; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:41435/solr/admin/collections
ok 82 post solr log into solr via cli # in 9659 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 27 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-05T12:33:31.584009819Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-05T12:33:31.584051678Z It should be set to 65000 to avoid operational disruption.
2026-05-05T12:33:31.584055345Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-05T12:33:31.584288437Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-05T12:33:31.584301693Z
2026-05-05T12:33:31.596881588Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-05T12:33:31.607232669Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-05T12:33:31.659564167Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-05T12:33:31.837140008Z WARNING: A command line option has enabled the Security Manager
2026-05-05T12:33:31.837165413Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-05T12:33:34.284986308Z 2026-05-05 12:33:34.281 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-05T12:33:34.820110624Z 2026-05-05 12:33:34.819 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-05T12:33:35.048203344Z 2026-05-05 12:33:35.047 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-05T12:33:35.108143329Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-05T12:33:35.108230315Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-05T12:33:35.108364184Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-05T12:33:35.108499321Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-05T12:33:35.107843696Z
2026-05-05T12:33:35.113076744Z 2026-05-05 12:33:35.112 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-05T12:33:35.198852266Z 2026-05-05 12:33:35.198 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-05T12:33:35.561594691Z 2026-05-05 12:33:35.560 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-05T12:33:35.561929919Z 2026-05-05 12:33:35.561 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-05T12:33:35.763093977Z 2026-05-05 12:33:35.762 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-05T12:33:37.485265547Z 2026-05-05 12:33:37.484 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-05T12:33:38.014655472Z 2026-05-05 12:33:38.013 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-05T12:33:38.026803667Z 2026-05-05 12:33:38.026 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-05T12:33:38.026918473Z 2026-05-05 12:33:38.026 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-05T12:33:38.035185110Z 2026-05-05 12:33:38.034 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-05T12:33:38.035200400Z 2026-05-05 12:33:38.034 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-05T12:33:38.138093548Z 2026-05-05 12:33:38.137 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-05T12:33:38.537712151Z 2026-05-05 12:33:38.536 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-05T12:33:38.604193287Z 2026-05-05 12:33:38.603 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-05T12:33:38.657979787Z 2026-05-05 12:33:38.657 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-05T12:33:38.686606510Z 2026-05-05 12:33:38.686 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-05T12:33:38.686710978Z 2026-05-05 12:33:38.686 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-05T12:33:38.808375606Z 2026-05-05 12:33:38.807 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-05T12:33:38.921267884Z 2026-05-05 12:33:38.920 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-05T12:33:38.933814622Z 2026-05-05 12:33:38.933 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-05T12:33:38.940183893Z 2026-05-05 12:33:38.939 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72081758103601152-solr-node1:8983_solr-n_0000000000) starting
2026-05-05T12:33:39.031452641Z 2026-05-05 12:33:39.030 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-05T12:33:39.034514967Z 2026-05-05 12:33:39.034 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-05T12:33:39.046680457Z 2026-05-05 12:33:39.046 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-05T12:33:39.149493064Z 2026-05-05 12:33:39.148 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-05T12:33:39.719179922Z 2026-05-05 12:33:39.718 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-05T12:33:39.974797535Z 2026-05-05 12:33:39.974 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-05T12:33:40.339821131Z 2026-05-05 12:33:40.339 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-05T12:33:40.854766401Z 2026-05-05 12:33:40.853 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-05T12:33:40.913402195Z 2026-05-05 12:33:40.912 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_05.request.log
2026-05-05T12:33:40.921489803Z 2026-05-05 12:33:40.921 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-05T12:33:40.923694266Z 2026-05-05 12:33:40.923 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @9317ms
2026-05-05T12:33:41.252879376Z 2026-05-05 12:33:41.251 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=106
2026-05-05T12:33:41.440707261Z 2026-05-05 12:33:41.440 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-05-05T12:33:46.117922103Z 2026-05-05 12:33:46.117 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-05-05T12:33:51.772947255Z 2026-05-05 12:33:51.771 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=10
2026-05-05T12:33:51.805334290Z 2026-05-05 12:33:51.804 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-05-05T12:33:51.849074753Z 2026-05-05 12:33:51.848 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=6
2026-05-05T12:33:52.174003526Z 2026-05-05 12:33:52.173 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-05T12:33:52.208303391Z 2026-05-05 12:33:52.205 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-05T12:33:52.234602153Z 2026-05-05 12:33:52.234 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:52.307293213Z 2026-05-05 12:33:52.305 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-05T12:33:52.307311854Z "core":"test-collection_shard3_replica_n1",
2026-05-05T12:33:52.307314683Z "node_name":"solr-node1:8983_solr",
2026-05-05T12:33:52.307316969Z "base_url":"http://solr-node1:8983/solr",
2026-05-05T12:33:52.307319095Z "collection":"test-collection",
2026-05-05T12:33:52.307321506Z "shard":"shard3",
2026-05-05T12:33:52.307323573Z "state":"down",
2026-05-05T12:33:52.307325567Z "type":"NRT",
2026-05-05T12:33:52.307327598Z "operation":"ADDREPLICA",
2026-05-05T12:33:52.307329632Z "waitForFinalState":"false"}
2026-05-05T12:33:52.316177217Z 2026-05-05 12:33:52.315 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-05T12:33:52.316193449Z "core":"test-collection_shard2_replica_n2",
2026-05-05T12:33:52.316196115Z "node_name":"solr-node3:8985_solr",
2026-05-05T12:33:52.316198633Z "base_url":"http://solr-node3:8985/solr",
2026-05-05T12:33:52.316201089Z "collection":"test-collection",
2026-05-05T12:33:52.316203686Z "shard":"shard2",
2026-05-05T12:33:52.316205698Z "state":"down",
2026-05-05T12:33:52.316207649Z "type":"NRT",
2026-05-05T12:33:52.316209505Z "operation":"ADDREPLICA",
2026-05-05T12:33:52.316211460Z "waitForFinalState":"false"}
2026-05-05T12:33:52.324382016Z 2026-05-05 12:33:52.323 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-05T12:33:52.324400737Z "core":"test-collection_shard1_replica_n4",
2026-05-05T12:33:52.324403851Z "node_name":"solr-node2:8984_solr",
2026-05-05T12:33:52.324406120Z "base_url":"http://solr-node2:8984/solr",
2026-05-05T12:33:52.324408265Z "collection":"test-collection",
2026-05-05T12:33:52.324420114Z "shard":"shard1",
2026-05-05T12:33:52.324422104Z "state":"down",
2026-05-05T12:33:52.324424090Z "type":"NRT",
2026-05-05T12:33:52.324426108Z "operation":"ADDREPLICA",
2026-05-05T12:33:52.324428059Z "waitForFinalState":"false"}
2026-05-05T12:33:52.436013239Z 2026-05-05 12:33:52.435 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:52.564764138Z 2026-05-05 12:33:52.564 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-05T12:33:52.655290669Z 2026-05-05 12:33:52.654 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-05T12:33:52.793151910Z 2026-05-05 12:33:52.792 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-05T12:33:53.537925071Z 2026-05-05 12:33:53.536 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-05T12:33:53.537954742Z 2026-05-05 12:33:53.536 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-05T12:33:53.598729449Z 2026-05-05 12:33:53.598 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-05T12:33:55.187998235Z 2026-05-05 12:33:55.186 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-05T12:33:55.266272537Z 2026-05-05 12:33:55.265 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-05T12:33:55.273034463Z 2026-05-05 12:33:55.272 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-05T12:33:55.274378800Z 2026-05-05 12:33:55.273 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-05T12:33:55.274404730Z 2026-05-05 12:33:55.273 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-05T12:33:55.399590219Z 2026-05-05 12:33:55.399 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-05T12:33:55.402620353Z 2026-05-05 12:33:55.402 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-05T12:33:55.404343586Z 2026-05-05 12:33:55.404 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-05T12:33:55.406135688Z 2026-05-05 12:33:55.405 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-05T12:33:55.426147214Z 2026-05-05 12:33:55.425 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-05T12:33:55.434407127Z 2026-05-05 12:33:55.434 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-05T12:33:55.443324287Z 2026-05-05 12:33:55.442 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-05T12:33:55.443344798Z 2026-05-05 12:33:55.442 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-05T12:33:55.494324615Z 2026-05-05 12:33:55.493 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-05T12:33:55.528246605Z 2026-05-05 12:33:55.527 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-05T12:33:55.530536143Z 2026-05-05 12:33:55.530 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-05T12:33:55.545696728Z 2026-05-05 12:33:55.545 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-05T12:33:55.545929918Z 2026-05-05 12:33:55.545 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-05T12:33:55.545938906Z 2026-05-05 12:33:55.545 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-05T12:33:55.547579392Z 2026-05-05 12:33:55.547 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-05T12:33:55.548108132Z 2026-05-05 12:33:55.547 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-05T12:33:55.549355596Z 2026-05-05 12:33:55.549 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72081758103601152-core_node3-n_0000000000
2026-05-05T12:33:55.571274648Z 2026-05-05 12:33:55.570 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-05T12:33:55.676205900Z 2026-05-05 12:33:55.675 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.679559102Z 2026-05-05 12:33:55.679 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-05T12:33:55.689795030Z 2026-05-05 12:33:55.688 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3126
2026-05-05T12:33:55.787826190Z 2026-05-05 12:33:55.787 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1774551,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-05T12:33:28.430541021Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-05T12:33:42.050242126Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-05T12:33:42.050269073Z It should be set to 65000 to avoid operational disruption.
2026-05-05T12:33:42.050272404Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-05T12:33:42.050561345Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-05T12:33:42.050568024Z
2026-05-05T12:33:42.059568683Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-05T12:33:42.068446824Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-05T12:33:42.123848174Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-05T12:33:42.166272549Z WARNING: A command line option has enabled the Security Manager
2026-05-05T12:33:42.166296744Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-05T12:33:43.667658403Z 2026-05-05 12:33:43.663 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-05T12:33:44.114279930Z 2026-05-05 12:33:44.113 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-05T12:33:44.159707748Z 2026-05-05 12:33:44.159 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-05T12:33:44.169476025Z 2026-05-05 12:33:44.168 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-05T12:33:44.169642551Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-05T12:33:44.169647367Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-05T12:33:44.169751825Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-05T12:33:44.169261315Z
2026-05-05T12:33:44.173641100Z 2026-05-05 12:33:44.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-05T12:33:44.207057916Z 2026-05-05 12:33:44.206 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-05T12:33:44.262042883Z 2026-05-05 12:33:44.261 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-05T12:33:44.511928875Z 2026-05-05 12:33:44.511 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-05T12:33:44.533367973Z 2026-05-05 12:33:44.532 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-05T12:33:44.533495490Z 2026-05-05 12:33:44.533 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-05T12:33:44.533606250Z 2026-05-05 12:33:44.533 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-05T12:33:44.659168940Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-05T12:33:44.659431281Z 2026-05-05 12:33:44.659 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-05T12:33:44.659657691Z 2026-05-05 12:33:44.659 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-05T12:33:44.716405761Z 2026-05-05 12:33:44.715 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-05T12:33:45.698245189Z 2026-05-05 12:33:45.696 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-05T12:33:45.988241865Z 2026-05-05 12:33:45.987 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-05T12:33:45.990121656Z 2026-05-05 12:33:45.987 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-05T12:33:46.005123064Z 2026-05-05 12:33:46.004 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-05T12:33:46.011261908Z 2026-05-05 12:33:46.010 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-05T12:33:46.014708837Z 2026-05-05 12:33:46.014 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-05T12:33:46.014797870Z 2026-05-05 12:33:46.014 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-05T12:33:46.049860664Z 2026-05-05 12:33:46.049 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-05T12:33:46.058765823Z 2026-05-05 12:33:46.058 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-05T12:33:46.082590393Z 2026-05-05 12:33:46.082 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-05T12:33:46.096758528Z 2026-05-05 12:33:46.096 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-05T12:33:46.107902383Z 2026-05-05 12:33:46.107 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-05T12:33:46.119121981Z 2026-05-05 12:33:46.118 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-05-05T12:33:46.184165428Z 2026-05-05 12:33:46.183 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-05T12:33:46.518706581Z 2026-05-05 12:33:46.518 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-05T12:33:46.765864920Z 2026-05-05 12:33:46.765 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-05T12:33:47.052887320Z 2026-05-05 12:33:47.052 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-05T12:33:47.208481243Z 2026-05-05 12:33:47.207 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-05T12:33:47.215611530Z 2026-05-05 12:33:47.215 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_05.request.log
2026-05-05T12:33:47.220410906Z 2026-05-05 12:33:47.219 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-05T12:33:47.222091548Z 2026-05-05 12:33:47.221 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5154ms
2026-05-05T12:33:47.445218144Z 2026-05-05 12:33:47.444 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=52
2026-05-05T12:33:47.519414694Z 2026-05-05 12:33:47.518 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19
2026-05-05T12:33:52.134005651Z 2026-05-05 12:33:52.133 INFO (qtp985324122-58-solr-node2-3) [c: s: r: x: t:solr-node2-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=7
2026-05-05T12:33:52.559835004Z 2026-05-05 12:33:52.559 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-05T12:33:52.638101261Z 2026-05-05 12:33:52.637 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-05T12:33:52.790386958Z 2026-05-05 12:33:52.787 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-05T12:33:53.291138006Z 2026-05-05 12:33:53.290 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-05T12:33:53.291240239Z 2026-05-05 12:33:53.290 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-05T12:33:53.322661246Z 2026-05-05 12:33:53.322 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-05T12:33:54.534884249Z 2026-05-05 12:33:54.534 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-05T12:33:54.556816460Z 2026-05-05 12:33:54.556 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-05T12:33:54.582171311Z 2026-05-05 12:33:54.581 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-05T12:33:54.583463926Z 2026-05-05 12:33:54.582 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-05T12:33:54.583476696Z 2026-05-05 12:33:54.583 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-05T12:33:54.662219240Z 2026-05-05 12:33:54.661 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-05T12:33:54.665143701Z 2026-05-05 12:33:54.664 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-05T12:33:54.667641873Z 2026-05-05 12:33:54.667 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-05T12:33:54.669590871Z 2026-05-05 12:33:54.669 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-05T12:33:54.689571198Z 2026-05-05 12:33:54.689 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-05T12:33:54.697359941Z 2026-05-05 12:33:54.696 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-05T12:33:54.742416168Z 2026-05-05 12:33:54.741 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-05T12:33:54.742438811Z 2026-05-05 12:33:54.742 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-05T12:33:54.751871798Z 2026-05-05 12:33:54.751 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-05T12:33:54.770093445Z 2026-05-05 12:33:54.769 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-05T12:33:54.771495727Z 2026-05-05 12:33:54.771 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-05T12:33:54.902634049Z 2026-05-05 12:33:54.902 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-05T12:33:54.902649720Z 2026-05-05 12:33:54.902 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-05T12:33:54.902653630Z 2026-05-05 12:33:54.902 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-05T12:33:54.903874832Z 2026-05-05 12:33:54.903 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-05T12:33:54.904379426Z 2026-05-05 12:33:54.904 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-05T12:33:54.905487751Z 2026-05-05 12:33:54.905 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72081758103601156-core_node6-n_0000000000
2026-05-05T12:33:54.921554244Z 2026-05-05 12:33:54.921 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-05T12:33:54.945126821Z 2026-05-05 12:33:54.944 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-05T12:33:54.954917005Z 2026-05-05 12:33:54.954 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2396
2026-05-05T12:33:55.056460824Z 2026-05-05 12:33:55.055 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 221] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.171882391Z 2026-05-05 12:33:55.171 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.676285910Z 2026-05-05 12:33:55.675 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.703247528Z 2026-05-05 12:33:55.702 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-05T12:33:55.787954961Z 2026-05-05 12:33:55.787 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.790582113Z 2026-05-05 12:33:55.790 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3638
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1774970,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-05T12:33:41.692958252Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-05T12:33:42.112913653Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-05T12:33:42.112950164Z It should be set to 65000 to avoid operational disruption.
2026-05-05T12:33:42.112953653Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-05T12:33:42.112956195Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-05T12:33:42.112958333Z
2026-05-05T12:33:42.121349576Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-05T12:33:42.129334172Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-05T12:33:42.178136899Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-05T12:33:42.212407487Z WARNING: A command line option has enabled the Security Manager
2026-05-05T12:33:42.212452904Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-05T12:33:43.687805801Z 2026-05-05 12:33:43.684 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-05T12:33:44.114235071Z 2026-05-05 12:33:44.113 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-05T12:33:44.159605082Z 2026-05-05 12:33:44.159 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-05T12:33:44.169512666Z 2026-05-05 12:33:44.168 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-05T12:33:44.169575922Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-05T12:33:44.169689265Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-05T12:33:44.169817107Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-05T12:33:44.169248944Z
2026-05-05T12:33:44.173474193Z 2026-05-05 12:33:44.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-05T12:33:44.207011432Z 2026-05-05 12:33:44.206 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-05T12:33:44.262053630Z 2026-05-05 12:33:44.261 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-05T12:33:44.511614367Z 2026-05-05 12:33:44.510 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-05T12:33:44.533052384Z 2026-05-05 12:33:44.532 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-05T12:33:44.533548974Z 2026-05-05 12:33:44.532 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-05T12:33:44.533556272Z 2026-05-05 12:33:44.532 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-05T12:33:44.659090295Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-05T12:33:44.659457860Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-05T12:33:44.659676594Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-05T12:33:44.716283785Z 2026-05-05 12:33:44.715 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-05T12:33:45.697339877Z 2026-05-05 12:33:45.696 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-05T12:33:45.988016327Z 2026-05-05 12:33:45.987 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-05T12:33:45.990184358Z 2026-05-05 12:33:45.987 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-05T12:33:46.003711004Z 2026-05-05 12:33:46.003 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-05T12:33:46.011235436Z 2026-05-05 12:33:46.010 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-05T12:33:46.014165625Z 2026-05-05 12:33:46.013 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-05T12:33:46.014232035Z 2026-05-05 12:33:46.013 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-05T12:33:46.049897924Z 2026-05-05 12:33:46.049 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-05T12:33:46.058610595Z 2026-05-05 12:33:46.058 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-05T12:33:46.082590395Z 2026-05-05 12:33:46.082 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-05T12:33:46.097708882Z 2026-05-05 12:33:46.097 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-05T12:33:46.108722831Z 2026-05-05 12:33:46.108 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-05T12:33:46.119119057Z 2026-05-05 12:33:46.118 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
2026-05-05T12:33:46.184229536Z 2026-05-05 12:33:46.183 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-05T12:33:46.518656135Z 2026-05-05 12:33:46.518 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-05T12:33:46.766608686Z 2026-05-05 12:33:46.766 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-05T12:33:47.052892106Z 2026-05-05 12:33:47.052 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-05T12:33:47.208522297Z 2026-05-05 12:33:47.207 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-05T12:33:47.215613078Z 2026-05-05 12:33:47.215 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_05.request.log
2026-05-05T12:33:47.220409921Z 2026-05-05 12:33:47.219 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-05T12:33:47.221678756Z 2026-05-05 12:33:47.221 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5093ms
2026-05-05T12:33:49.876359989Z 2026-05-05 12:33:49.867 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=112
2026-05-05T12:33:49.974867288Z 2026-05-05 12:33:49.974 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19
2026-05-05T12:33:52.561046321Z 2026-05-05 12:33:52.560 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-05T12:33:52.647671509Z 2026-05-05 12:33:52.647 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-05T12:33:52.790974166Z 2026-05-05 12:33:52.790 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-05T12:33:53.293331040Z 2026-05-05 12:33:53.292 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-05T12:33:53.293389425Z 2026-05-05 12:33:53.293 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-05T12:33:53.322700609Z 2026-05-05 12:33:53.322 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-05T12:33:54.535078961Z 2026-05-05 12:33:54.534 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-05T12:33:54.557772146Z 2026-05-05 12:33:54.557 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-05T12:33:54.582252463Z 2026-05-05 12:33:54.581 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-05T12:33:54.583529351Z 2026-05-05 12:33:54.582 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-05T12:33:54.583633944Z 2026-05-05 12:33:54.583 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-05T12:33:54.662546932Z 2026-05-05 12:33:54.662 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-05T12:33:54.665884116Z 2026-05-05 12:33:54.665 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-05T12:33:54.667901185Z 2026-05-05 12:33:54.667 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-05T12:33:54.669615933Z 2026-05-05 12:33:54.669 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-05T12:33:54.689633448Z 2026-05-05 12:33:54.689 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-05T12:33:54.697667836Z 2026-05-05 12:33:54.697 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-05T12:33:54.742952247Z 2026-05-05 12:33:54.742 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-05T12:33:54.743024874Z 2026-05-05 12:33:54.742 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-05T12:33:54.753287812Z 2026-05-05 12:33:54.752 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-05T12:33:54.877866818Z 2026-05-05 12:33:54.877 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-05T12:33:54.879383191Z 2026-05-05 12:33:54.879 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-05T12:33:54.900867084Z 2026-05-05 12:33:54.900 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-05T12:33:54.900888973Z 2026-05-05 12:33:54.900 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-05T12:33:54.900988089Z 2026-05-05 12:33:54.900 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-05T12:33:54.902220486Z 2026-05-05 12:33:54.901 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-05T12:33:54.902803347Z 2026-05-05 12:33:54.902 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-05T12:33:54.903994841Z 2026-05-05 12:33:54.903 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72081758103601155-core_node5-n_0000000000
2026-05-05T12:33:54.922396562Z 2026-05-05 12:33:54.921 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-05T12:33:55.056433902Z 2026-05-05 12:33:55.055 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 221] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.062912516Z 2026-05-05 12:33:55.062 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-05T12:33:55.071403935Z 2026-05-05 12:33:55.070 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2512
2026-05-05T12:33:55.172324084Z 2026-05-05 12:33:55.171 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.676366589Z 2026-05-05 12:33:55.675 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-05T12:33:55.787975505Z 2026-05-05 12:33:55.787 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1775100,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-05T12:33:41.916923496Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 34097 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# 1cdb71aca53d: Pulling fs layer
# 0d8eec1b6c24: Pulling fs layer
# 7e69c308213e: Pulling fs layer
# c8f61c25585a: Pulling fs layer
# d959d69295ca: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# c8f61c25585a: Waiting
# d959d69295ca: Waiting
# 4f4fb700ef54: Waiting
# 0d8eec1b6c24: Download complete
# 7e69c308213e: Download complete
# d959d69295ca: Verifying Checksum
# d959d69295ca: Download complete
# c8f61c25585a: Verifying Checksum
# c8f61c25585a: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 1cdb71aca53d: Verifying Checksum
# 1cdb71aca53d: Download complete
# 1cdb71aca53d: Pull complete
# 0d8eec1b6c24: Pull complete
# 7e69c308213e: Pull complete
# c8f61c25585a: Pull complete
# d959d69295ca: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:dd9c7a5a0d1a56548540e76fb46ce879d58198257ba5caf6de6f48bb614155a7
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# a9bd7bdedbc5b8c1102990d9bf526d83db6e5fa98bc415b5d72dc5b454ed444c
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 71bd2354e93a912a99d6f4ae12368e6ecd861f38776f1bb8511737bdd1126e9d
# Starting solr-node2 connected to first node's ZooKeeper
# a39e7337abc206741b456c80f667aa2ec0c1e5014063d5538cd922713424a112
# Started solr-node3 connected to first node's ZooKeeper
# e7efbdb0ecaa3c6c41c1aec46d30b38782608fb23e362966d848ba53fd0c36c1
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-05 12:33:51.901; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-05 12:33:51.952; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node2:8984/solr] to [http://solr-node2:8984/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 15669 ms
ok 85 snapshot lifecycle # in 4839 ms
ok 86 snapshot list # in 3241 ms
ok 87 snapshot describe # in 3391 ms
ok 88 start solr with ssl # in 16152 ms
ok 89 use different hostname when not checking peer-name # in 24719 ms
ok 90 start solr with ssl and auth # in 14579 ms
ok 91 start solr with client truststore and security manager # in 16260 ms
ok 92 start solr with mTLS needed # in 46613 ms
ok 93 start solr with mTLS wanted # in 34118 ms
ok 94 test keystore reload # in 58704 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15882 ms
ok 96 stop command for single port # in 18371 ms
ok 97 check stop command doesn't hang # in 218 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7952 ms
ok 99 deprecated system properties converted to modern properties # in 195160 ms
ok 100 start with custom jetty options # in 8033 ms
ok 101 bootstrapping a configset # in 9206 ms
ok 102 status detects locally running solr # in 9583 ms
ok 103 status with --solr-url from user # in 8064 ms
ok 104 status with --port from user # in 7924 ms
ok 105 multiple connection options are prevented # in 841 ms
ok 106 status with invalid --solr-url from user # in 1258 ms
ok 107 status with --short format # in 195843 ms
ok 108 searching solr via locally executed streaming expression # in 1743 ms
ok 109 searching solr via remotely executed streaming expression # in 1314 ms
ok 110 variable interpolation # in 1689 ms
ok 111 searching solr without credentials fails with error # in 1135 ms
ok 112 --version returns Solr version # in 759 ms
ok 113 version as direct tool call still runs # in 743 ms
ok 114 short help # in 821 ms
ok 115 short help is inferred # in 815 ms
ok 116 long help # in 791 ms
ok 117 running subcommands with zk is prevented # in 749 ms
ok 118 listing out files # in 2210 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7861 ms
ok 120 copying files around # in 11207 ms
ok 121 upconfig # in 2537 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1209 ms
ok 123 downconfig # in 1179 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1450 ms
ok 125 env var ZK_HOST is honored # in 3168 ms
ok 126 Cleanup # in 1863 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 38s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/yf4oio33rw32e
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 199 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/199/
No tests ran.
Build Log:
[...truncated 425 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:45211/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45211.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45211.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:45211/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:45211 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-04 12:28:32.998; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-04 12:28:33.002; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:45211/solr/admin/collections
not ok 81 verbose echo the Solr response # in 1061 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45211.
#
# ERROR: Server refused connection at: http://localhost:45211/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45211.
#
# ERROR: Server refused connection at: http://localhost:45211/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8482 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 8 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 8 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 13 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-04T12:29:01.695059827Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-04T12:29:01.695087871Z It should be set to 65000 to avoid operational disruption.
2026-05-04T12:29:01.695090911Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-04T12:29:01.695373331Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-04T12:29:01.695388947Z
2026-05-04T12:29:01.704079563Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-04T12:29:01.711346050Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-04T12:29:01.754577002Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-04T12:29:01.784979713Z WARNING: A command line option has enabled the Security Manager
2026-05-04T12:29:01.784999003Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-04T12:29:03.033443005Z 2026-05-04 12:29:03.030 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-04T12:29:03.284713185Z 2026-05-04 12:29:03.284 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-04T12:29:03.308330004Z 2026-05-04 12:29:03.307 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-04T12:29:03.314699914Z 2026-05-04 12:29:03.314 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-04T12:29:03.314773283Z 2026-05-04 12:29:03.314 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-04T12:29:03.314878003Z 2026-05-04 12:29:03.314 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-04T12:29:03.314985320Z 2026-05-04 12:29:03.314 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-04T12:29:03.314461818Z
2026-05-04T12:29:03.317263327Z 2026-05-04 12:29:03.316 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-04T12:29:03.337437614Z 2026-05-04 12:29:03.336 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-04T12:29:03.349907085Z 2026-05-04 12:29:03.349 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-04T12:29:03.349921030Z 2026-05-04 12:29:03.349 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-04T12:29:03.382741136Z 2026-05-04 12:29:03.382 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-04T12:29:04.136143021Z 2026-05-04 12:29:04.135 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-04T12:29:04.354620734Z 2026-05-04 12:29:04.354 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-04T12:29:04.359134607Z 2026-05-04 12:29:04.358 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-04T12:29:04.359257232Z 2026-05-04 12:29:04.358 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-04T12:29:04.361616724Z 2026-05-04 12:29:04.361 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-04T12:29:04.361627954Z 2026-05-04 12:29:04.361 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-04T12:29:04.395003450Z 2026-05-04 12:29:04.394 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-04T12:29:04.864515400Z 2026-05-04 12:29:04.863 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-04T12:29:04.875934534Z 2026-05-04 12:29:04.875 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-04T12:29:04.909372831Z 2026-05-04 12:29:04.908 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-04T12:29:05.003106853Z 2026-05-04 12:29:05.002 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-04T12:29:05.003213728Z 2026-05-04 12:29:05.002 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-04T12:29:05.115206300Z 2026-05-04 12:29:05.114 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-04T12:29:05.139897671Z 2026-05-04 12:29:05.139 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-04T12:29:05.151565752Z 2026-05-04 12:29:05.151 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-04T12:29:05.157135423Z 2026-05-04 12:29:05.156 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72075992125079552-solr-node1:8983_solr-n_0000000000) starting
2026-05-04T12:29:05.220078225Z 2026-05-04 12:29:05.219 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-04T12:29:05.220825037Z 2026-05-04 12:29:05.220 INFO (OverseerStateUpdate-72075992125079552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-04T12:29:05.230206537Z 2026-05-04 12:29:05.229 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-04T12:29:05.271615390Z 2026-05-04 12:29:05.271 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-04T12:29:05.620453433Z 2026-05-04 12:29:05.619 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-04T12:29:05.785301493Z 2026-05-04 12:29:05.784 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-04T12:29:06.037621786Z 2026-05-04 12:29:06.037 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-04T12:29:06.165490489Z 2026-05-04 12:29:06.164 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-04T12:29:06.171141043Z 2026-05-04 12:29:06.170 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_04.request.log
2026-05-04T12:29:06.175434499Z 2026-05-04 12:29:06.175 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-04T12:29:06.200779911Z 2026-05-04 12:29:06.176 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4466ms
2026-05-04T12:29:06.356484037Z 2026-05-04 12:29:06.355 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-05-04T12:29:06.406553944Z 2026-05-04 12:29:06.405 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=11
2026-05-04T12:29:09.341425928Z 2026-05-04 12:29:09.341 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-04T12:29:09.964746670Z 2026-05-04 12:29:09.964 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-04T12:29:12.825742646Z 2026-05-04 12:29:12.825 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-05-04T12:29:12.850059116Z 2026-05-04 12:29:12.849 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=7
2026-05-04T12:29:12.873858047Z 2026-05-04 12:29:12.873 INFO (qtp985324122-100-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-05-04T12:29:13.111540759Z 2026-05-04 12:29:13.109 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-04T12:29:13.127671471Z 2026-05-04 12:29:13.127 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-04T12:29:13.135275601Z 2026-05-04 12:29:13.134 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:13.169663298Z 2026-05-04 12:29:13.169 INFO (OverseerStateUpdate-72075992125079552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-04T12:29:13.169682923Z "core":"test-collection_shard3_replica_n1",
2026-05-04T12:29:13.169685802Z "node_name":"solr-node1:8983_solr",
2026-05-04T12:29:13.169688264Z "base_url":"http://solr-node1:8983/solr",
2026-05-04T12:29:13.169690485Z "collection":"test-collection",
2026-05-04T12:29:13.169692565Z "shard":"shard3",
2026-05-04T12:29:13.169695096Z "state":"down",
2026-05-04T12:29:13.169697136Z "type":"NRT",
2026-05-04T12:29:13.169699099Z "operation":"ADDREPLICA",
2026-05-04T12:29:13.169701259Z "waitForFinalState":"false"}
2026-05-04T12:29:13.174956618Z 2026-05-04 12:29:13.174 INFO (OverseerStateUpdate-72075992125079552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-04T12:29:13.174973449Z "core":"test-collection_shard2_replica_n2",
2026-05-04T12:29:13.174976407Z "node_name":"solr-node3:8985_solr",
2026-05-04T12:29:13.174978486Z "base_url":"http://solr-node3:8985/solr",
2026-05-04T12:29:13.174980511Z "collection":"test-collection",
2026-05-04T12:29:13.174982666Z "shard":"shard2",
2026-05-04T12:29:13.174984683Z "state":"down",
2026-05-04T12:29:13.174986627Z "type":"NRT",
2026-05-04T12:29:13.174988721Z "operation":"ADDREPLICA",
2026-05-04T12:29:13.174990673Z "waitForFinalState":"false"}
2026-05-04T12:29:13.182929529Z 2026-05-04 12:29:13.182 INFO (OverseerStateUpdate-72075992125079552-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-04T12:29:13.182943705Z "core":"test-collection_shard1_replica_n4",
2026-05-04T12:29:13.182946158Z "node_name":"solr-node2:8984_solr",
2026-05-04T12:29:13.182957682Z "base_url":"http://solr-node2:8984/solr",
2026-05-04T12:29:13.182959783Z "collection":"test-collection",
2026-05-04T12:29:13.182961925Z "shard":"shard1",
2026-05-04T12:29:13.182963853Z "state":"down",
2026-05-04T12:29:13.182965710Z "type":"NRT",
2026-05-04T12:29:13.182967634Z "operation":"ADDREPLICA",
2026-05-04T12:29:13.182969607Z "waitForFinalState":"false"}
2026-05-04T12:29:13.291152698Z 2026-05-04 12:29:13.290 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:13.347359192Z 2026-05-04 12:29:13.346 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-04T12:29:13.391663750Z 2026-05-04 12:29:13.391 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-04T12:29:13.438853639Z 2026-05-04 12:29:13.438 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-04T12:29:13.710322892Z 2026-05-04 12:29:13.709 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-04T12:29:13.710408946Z 2026-05-04 12:29:13.709 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-04T12:29:13.730959179Z 2026-05-04 12:29:13.730 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-04T12:29:14.361495195Z 2026-05-04 12:29:14.360 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-04T12:29:14.394684167Z 2026-05-04 12:29:14.394 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-04T12:29:14.401025090Z 2026-05-04 12:29:14.400 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-04T12:29:14.401903925Z 2026-05-04 12:29:14.401 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-04T12:29:14.401978435Z 2026-05-04 12:29:14.401 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-04T12:29:14.467765171Z 2026-05-04 12:29:14.466 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-04T12:29:14.469886175Z 2026-05-04 12:29:14.469 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-04T12:29:14.471615695Z 2026-05-04 12:29:14.471 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-04T12:29:14.473000113Z 2026-05-04 12:29:14.472 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-04T12:29:14.487671928Z 2026-05-04 12:29:14.487 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-04T12:29:14.496750308Z 2026-05-04 12:29:14.496 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-04T12:29:14.502629933Z 2026-05-04 12:29:14.502 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-04T12:29:14.502701575Z 2026-05-04 12:29:14.502 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-04T12:29:14.513999825Z 2026-05-04 12:29:14.513 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-04T12:29:14.520417670Z 2026-05-04 12:29:14.519 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-04T12:29:14.521744951Z 2026-05-04 12:29:14.521 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-04T12:29:14.533219009Z 2026-05-04 12:29:14.532 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-04T12:29:14.533290730Z 2026-05-04 12:29:14.533 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-04T12:29:14.533322608Z 2026-05-04 12:29:14.533 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-04T12:29:14.534286145Z 2026-05-04 12:29:14.534 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-04T12:29:14.534741796Z 2026-05-04 12:29:14.534 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-04T12:29:14.535674944Z 2026-05-04 12:29:14.535 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72075992125079552-core_node3-n_0000000000
2026-05-04T12:29:14.540754580Z 2026-05-04 12:29:14.540 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-04T12:29:14.644734703Z 2026-05-04 12:29:14.644 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:14.648644636Z 2026-05-04 12:29:14.648 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-04T12:29:14.654872155Z 2026-05-04 12:29:14.654 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1309
2026-05-04T12:29:14.756030785Z 2026-05-04 12:29:14.755 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 969962,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-04T12:29:01.490682464Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-04T12:29:06.675927924Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-04T12:29:06.675949204Z It should be set to 65000 to avoid operational disruption.
2026-05-04T12:29:06.675952368Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-04T12:29:06.676282968Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-04T12:29:06.676295882Z
2026-05-04T12:29:06.685455379Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-04T12:29:06.693341685Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-04T12:29:06.736239383Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-04T12:29:06.769515634Z WARNING: A command line option has enabled the Security Manager
2026-05-04T12:29:06.769532285Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-04T12:29:07.782936803Z 2026-05-04 12:29:07.779 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-04T12:29:08.037534976Z 2026-05-04 12:29:08.037 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-04T12:29:08.061993772Z 2026-05-04 12:29:08.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-04T12:29:08.068910681Z 2026-05-04 12:29:08.068 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-04T12:29:08.068952544Z 2026-05-04 12:29:08.068 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-04T12:29:08.069065155Z 2026-05-04 12:29:08.068 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-04T12:29:08.069193794Z 2026-05-04 12:29:08.068 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-04T12:29:08.068665812Z
2026-05-04T12:29:08.071796646Z 2026-05-04 12:29:08.071 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-04T12:29:08.093530470Z 2026-05-04 12:29:08.093 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-04T12:29:08.117628184Z 2026-05-04 12:29:08.117 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-04T12:29:08.249523834Z 2026-05-04 12:29:08.249 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-04T12:29:08.266166112Z 2026-05-04 12:29:08.265 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-04T12:29:08.266235356Z 2026-05-04 12:29:08.265 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-04T12:29:08.266343673Z 2026-05-04 12:29:08.265 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-04T12:29:08.375643063Z 2026-05-04 12:29:08.375 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-04T12:29:08.375781413Z 2026-05-04 12:29:08.375 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-04T12:29:08.375884552Z 2026-05-04 12:29:08.375 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-04T12:29:08.409014560Z 2026-05-04 12:29:08.408 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-04T12:29:09.067831927Z 2026-05-04 12:29:09.067 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-04T12:29:09.234968827Z 2026-05-04 12:29:09.233 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-04T12:29:09.236342566Z 2026-05-04 12:29:09.234 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-04T12:29:09.246498529Z 2026-05-04 12:29:09.246 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-04T12:29:09.252415949Z 2026-05-04 12:29:09.252 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-04T12:29:09.256970088Z 2026-05-04 12:29:09.256 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-04T12:29:09.256983171Z 2026-05-04 12:29:09.256 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-04T12:29:09.284443842Z 2026-05-04 12:29:09.284 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-04T12:29:09.291707159Z 2026-05-04 12:29:09.291 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-04T12:29:09.311371123Z 2026-05-04 12:29:09.310 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-04T12:29:09.322841900Z 2026-05-04 12:29:09.322 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-04T12:29:09.332346224Z 2026-05-04 12:29:09.331 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-04T12:29:09.342370841Z 2026-05-04 12:29:09.342 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-04T12:29:09.383240415Z 2026-05-04 12:29:09.382 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-04T12:29:09.661448735Z 2026-05-04 12:29:09.660 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-04T12:29:09.830971108Z 2026-05-04 12:29:09.830 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-04T12:29:09.965896264Z 2026-05-04 12:29:09.965 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-04T12:29:10.078208401Z 2026-05-04 12:29:10.077 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-04T12:29:10.210370919Z 2026-05-04 12:29:10.209 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-04T12:29:10.216634159Z 2026-05-04 12:29:10.216 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_04.request.log
2026-05-04T12:29:10.220873777Z 2026-05-04 12:29:10.220 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-04T12:29:10.222443854Z 2026-05-04 12:29:10.222 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3530ms
2026-05-04T12:29:10.375944387Z 2026-05-04 12:29:10.375 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=42
2026-05-04T12:29:10.432154659Z 2026-05-04 12:29:10.431 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-04T12:29:13.345892800Z 2026-05-04 12:29:13.345 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-04T12:29:13.395248468Z 2026-05-04 12:29:13.394 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-04T12:29:13.448993124Z 2026-05-04 12:29:13.448 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-04T12:29:13.715035730Z 2026-05-04 12:29:13.714 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-04T12:29:13.715059996Z 2026-05-04 12:29:13.714 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-04T12:29:13.732344155Z 2026-05-04 12:29:13.731 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-04T12:29:14.133761472Z 2026-05-04 12:29:14.133 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-04T12:29:14.157880562Z 2026-05-04 12:29:14.157 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-04T12:29:14.162872258Z 2026-05-04 12:29:14.162 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-04T12:29:14.164266798Z 2026-05-04 12:29:14.163 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-04T12:29:14.164281276Z 2026-05-04 12:29:14.163 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-04T12:29:14.239169062Z 2026-05-04 12:29:14.238 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-04T12:29:14.242237336Z 2026-05-04 12:29:14.241 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-04T12:29:14.244528377Z 2026-05-04 12:29:14.244 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-04T12:29:14.245980436Z 2026-05-04 12:29:14.245 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-04T12:29:14.257189816Z 2026-05-04 12:29:14.256 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-04T12:29:14.264546956Z 2026-05-04 12:29:14.264 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-04T12:29:14.269405474Z 2026-05-04 12:29:14.269 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-04T12:29:14.269434581Z 2026-05-04 12:29:14.269 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-04T12:29:14.275876654Z 2026-05-04 12:29:14.275 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-04T12:29:14.284123814Z 2026-05-04 12:29:14.283 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-04T12:29:14.285366878Z 2026-05-04 12:29:14.285 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-04T12:29:14.305390720Z 2026-05-04 12:29:14.305 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-04T12:29:14.305526955Z 2026-05-04 12:29:14.305 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-04T12:29:14.305605182Z 2026-05-04 12:29:14.305 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-04T12:29:14.306736176Z 2026-05-04 12:29:14.306 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-04T12:29:14.308659136Z 2026-05-04 12:29:14.308 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-04T12:29:14.309805037Z 2026-05-04 12:29:14.309 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72075992125079555-core_node6-n_0000000000
2026-05-04T12:29:14.319524572Z 2026-05-04 12:29:14.319 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-04T12:29:14.461228959Z 2026-05-04 12:29:14.460 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:14.465208396Z 2026-05-04 12:29:14.464 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-04T12:29:14.473549911Z 2026-05-04 12:29:14.473 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1129
2026-05-04T12:29:14.644921964Z 2026-05-04 12:29:14.644 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:14.756252441Z 2026-05-04 12:29:14.755 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 970360,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-04T12:29:06.487599282Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-04T12:29:06.875578785Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-04T12:29:06.875608797Z It should be set to 65000 to avoid operational disruption.
2026-05-04T12:29:06.875612108Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-04T12:29:06.875893707Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-04T12:29:06.875899488Z
2026-05-04T12:29:06.884919288Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-04T12:29:06.892435209Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-04T12:29:06.939411116Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-04T12:29:06.968816990Z WARNING: A command line option has enabled the Security Manager
2026-05-04T12:29:06.968837723Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-04T12:29:08.309327599Z 2026-05-04 12:29:08.306 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-04T12:29:08.569046544Z 2026-05-04 12:29:08.568 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-04T12:29:08.593288847Z 2026-05-04 12:29:08.592 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-04T12:29:08.600345618Z 2026-05-04 12:29:08.599 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-04T12:29:08.600371046Z 2026-05-04 12:29:08.599 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-04T12:29:08.600504197Z 2026-05-04 12:29:08.600 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-04T12:29:08.600614956Z 2026-05-04 12:29:08.600 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-04T12:29:08.600088103Z
2026-05-04T12:29:08.603232392Z 2026-05-04 12:29:08.602 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-04T12:29:08.625201678Z 2026-05-04 12:29:08.624 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-04T12:29:08.650466742Z 2026-05-04 12:29:08.649 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-04T12:29:08.783319641Z 2026-05-04 12:29:08.782 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-04T12:29:08.799301249Z 2026-05-04 12:29:08.798 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-04T12:29:08.799381704Z 2026-05-04 12:29:08.799 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-04T12:29:08.799492387Z 2026-05-04 12:29:08.799 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-04T12:29:08.907833976Z 2026-05-04 12:29:08.907 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-04T12:29:08.907855720Z 2026-05-04 12:29:08.907 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-04T12:29:08.907922007Z 2026-05-04 12:29:08.907 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-04T12:29:08.940049845Z 2026-05-04 12:29:08.939 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-04T12:29:09.721046470Z 2026-05-04 12:29:09.720 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-04T12:29:09.856444348Z 2026-05-04 12:29:09.856 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-04T12:29:09.857779156Z 2026-05-04 12:29:09.856 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-04T12:29:09.871511602Z 2026-05-04 12:29:09.871 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-04T12:29:09.877776364Z 2026-05-04 12:29:09.877 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-04T12:29:09.881861909Z 2026-05-04 12:29:09.881 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-04T12:29:09.881940527Z 2026-05-04 12:29:09.881 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-04T12:29:09.911244767Z 2026-05-04 12:29:09.910 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-04T12:29:09.918847748Z 2026-05-04 12:29:09.918 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-04T12:29:09.937993573Z 2026-05-04 12:29:09.937 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-04T12:29:09.948517192Z 2026-05-04 12:29:09.948 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-04T12:29:09.957468419Z 2026-05-04 12:29:09.957 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-04T12:29:09.965486850Z 2026-05-04 12:29:09.965 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-04T12:29:10.003141403Z 2026-05-04 12:29:10.002 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-04T12:29:10.250109784Z 2026-05-04 12:29:10.249 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-04T12:29:10.411385474Z 2026-05-04 12:29:10.410 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-04T12:29:10.654892104Z 2026-05-04 12:29:10.654 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-04T12:29:10.772671623Z 2026-05-04 12:29:10.771 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-04T12:29:10.778261624Z 2026-05-04 12:29:10.777 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_04.request.log
2026-05-04T12:29:10.782679698Z 2026-05-04 12:29:10.782 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-04T12:29:10.783383692Z 2026-05-04 12:29:10.783 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3892ms
2026-05-04T12:29:11.761426235Z 2026-05-04 12:29:11.757 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=66
2026-05-04T12:29:11.823303750Z 2026-05-04 12:29:11.822 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-05-04T12:29:13.084189863Z 2026-05-04 12:29:13.083 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=10
2026-05-04T12:29:13.346290889Z 2026-05-04 12:29:13.345 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-04T12:29:13.417737804Z 2026-05-04 12:29:13.417 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-04T12:29:13.478510330Z 2026-05-04 12:29:13.478 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-04T12:29:13.746840061Z 2026-05-04 12:29:13.746 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-04T12:29:13.746871583Z 2026-05-04 12:29:13.746 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-04T12:29:13.764764329Z 2026-05-04 12:29:13.764 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-04T12:29:14.196264042Z 2026-05-04 12:29:14.195 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-04T12:29:14.219615519Z 2026-05-04 12:29:14.219 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-04T12:29:14.225181296Z 2026-05-04 12:29:14.224 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-04T12:29:14.226028172Z 2026-05-04 12:29:14.225 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-04T12:29:14.226084224Z 2026-05-04 12:29:14.225 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-04T12:29:14.267000415Z 2026-05-04 12:29:14.266 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-04T12:29:14.269502461Z 2026-05-04 12:29:14.269 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-04T12:29:14.271767039Z 2026-05-04 12:29:14.271 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-04T12:29:14.273217909Z 2026-05-04 12:29:14.272 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-04T12:29:14.284736270Z 2026-05-04 12:29:14.284 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-04T12:29:14.292149181Z 2026-05-04 12:29:14.291 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-04T12:29:14.301714277Z 2026-05-04 12:29:14.301 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-04T12:29:14.301775067Z 2026-05-04 12:29:14.301 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-04T12:29:14.308314177Z 2026-05-04 12:29:14.307 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-04T12:29:14.321465447Z 2026-05-04 12:29:14.320 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-04T12:29:14.322769524Z 2026-05-04 12:29:14.322 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-04T12:29:14.333898867Z 2026-05-04 12:29:14.333 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-04T12:29:14.333920923Z 2026-05-04 12:29:14.333 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-04T12:29:14.334135815Z 2026-05-04 12:29:14.333 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-04T12:29:14.335195846Z 2026-05-04 12:29:14.334 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-04T12:29:14.335752462Z 2026-05-04 12:29:14.335 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-04T12:29:14.336725977Z 2026-05-04 12:29:14.336 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72075992125079556-core_node5-n_0000000000
2026-05-04T12:29:14.343972243Z 2026-05-04 12:29:14.343 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-04T12:29:14.461192373Z 2026-05-04 12:29:14.460 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:14.465075318Z 2026-05-04 12:29:14.464 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-04T12:29:14.473016679Z 2026-05-04 12:29:14.472 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1128
2026-05-04T12:29:14.644897286Z 2026-05-04 12:29:14.644 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:14.665673864Z 2026-05-04 12:29:14.665 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-04T12:29:14.756182234Z 2026-05-04 12:29:14.755 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-04T12:29:14.757898472Z 2026-05-04 12:29:14.757 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=1662
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 970545,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-04T12:29:06.672673428Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 24550 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Pulling fs layer
# 601f2c23751f: Waiting
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# 0ed719aef393: Waiting
# c937aee9a870: Pulling fs layer
# 69a09c96e8bf: Waiting
# 9cdfe5320de8: Pulling fs layer
# c937aee9a870: Waiting
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# e58d4eb27818: Pulling fs layer
# f567f46ba2b1: Pulling fs layer
# 98ff47d1ea92: Pulling fs layer
# 6515dccb230c: Pulling fs layer
# a1269c0ecd80: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 6515dccb230c: Waiting
# a1269c0ecd80: Waiting
# 4f4fb700ef54: Waiting
# f567f46ba2b1: Verifying Checksum
# f567f46ba2b1: Download complete
# 6515dccb230c: Download complete
# 98ff47d1ea92: Verifying Checksum
# 98ff47d1ea92: Download complete
# 4f4fb700ef54: Download complete
# a1269c0ecd80: Download complete
# e58d4eb27818: Verifying Checksum
# e58d4eb27818: Download complete
# e58d4eb27818: Pull complete
# f567f46ba2b1: Pull complete
# 98ff47d1ea92: Pull complete
# 6515dccb230c: Pull complete
# a1269c0ecd80: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:93831118a29de48ba0e6c3aa3c158973957b6d4fed14aaf73055cbdf080fa38b
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 61345c3a2112ccf9a60ba4e9d9d51aa744fac018c6b6d96270393f29a503fba3
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 584bfe0083eb9ac40a4cc70f9d3f03db469052c20c5f8aaccd17e4016a915b62
# Starting solr-node2 connected to first node's ZooKeeper
# 360ffbe463dac265be977ae60d44f60a5dc8e5b02448bda469b4e73b5e237be1
# Started solr-node3 connected to first node's ZooKeeper
# a3d5971a6971cf0c6f14732cb7556f7208c727cbac1621c313326f7c382806a0
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-04 12:29:12.900; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-04 12:29:12.929; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14417 ms
ok 85 snapshot lifecycle # in 4423 ms
ok 86 snapshot list # in 3071 ms
ok 87 snapshot describe # in 3071 ms
ok 88 start solr with ssl # in 15612 ms
ok 89 use different hostname when not checking peer-name # in 23118 ms
ok 90 start solr with ssl and auth # in 13803 ms
ok 91 start solr with client truststore and security manager # in 15372 ms
ok 92 start solr with mTLS needed # in 45833 ms
ok 93 start solr with mTLS wanted # in 33472 ms
ok 94 test keystore reload # in 55923 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15480 ms
ok 96 stop command for single port # in 17597 ms
ok 97 check stop command doesn't hang # in 208 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7943 ms
ok 99 deprecated system properties converted to modern properties # in 195033 ms
ok 100 start with custom jetty options # in 7667 ms
ok 101 bootstrapping a configset # in 8870 ms
ok 102 status detects locally running solr # in 8903 ms
ok 103 status with --solr-url from user # in 7605 ms
ok 104 status with --port from user # in 7661 ms
ok 105 multiple connection options are prevented # in 800 ms
ok 106 status with invalid --solr-url from user # in 1132 ms
ok 107 status with --short format # in 195792 ms
ok 108 searching solr via locally executed streaming expression # in 1595 ms
ok 109 searching solr via remotely executed streaming expression # in 1169 ms
ok 110 variable interpolation # in 1569 ms
ok 111 searching solr without credentials fails with error # in 1040 ms
ok 112 --version returns Solr version # in 692 ms
ok 113 version as direct tool call still runs # in 709 ms
ok 114 short help # in 732 ms
ok 115 short help is inferred # in 735 ms
ok 116 long help # in 731 ms
ok 117 running subcommands with zk is prevented # in 685 ms
ok 118 listing out files # in 2068 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7397 ms
ok 120 copying files around # in 10423 ms
ok 121 upconfig # in 2440 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1037 ms
ok 123 downconfig # in 1097 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1343 ms
ok 125 env var ZK_HOST is honored # in 3057 ms
ok 126 Cleanup # in 1801 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 17s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/dtb3uxi77jwuw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 198 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/198/
No tests ran.
Build Log:
[...truncated 425 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:36477/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:36477.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:36477.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:36477/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:36477 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-03 13:51:45.811; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-03 13:51:45.814; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:36477/solr/admin/collections
not ok 81 verbose echo the Solr response # in 1097 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:36477.
#
# ERROR: Server refused connection at: http://localhost:36477/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:36477.
#
# ERROR: Server refused connection at: http://localhost:36477/solr/admin/info/system
ok 82 post solr log into solr via cli # in 9089 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-03T13:52:11.472486669Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-03T13:52:11.472528656Z It should be set to 65000 to avoid operational disruption.
2026-05-03T13:52:11.472531852Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-03T13:52:11.472764173Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-03T13:52:11.472789466Z
2026-05-03T13:52:11.482327127Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-03T13:52:11.489541145Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-03T13:52:11.548149630Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-03T13:52:11.582597037Z WARNING: A command line option has enabled the Security Manager
2026-05-03T13:52:11.582623287Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-03T13:52:13.167800129Z 2026-05-03 13:52:13.163 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-03T13:52:13.436818129Z 2026-05-03 13:52:13.436 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-03T13:52:13.476991278Z 2026-05-03 13:52:13.476 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-03T13:52:13.484625356Z 2026-05-03 13:52:13.484 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-03T13:52:13.484692044Z 2026-05-03 13:52:13.484 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-03T13:52:13.484856830Z 2026-05-03 13:52:13.484 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-03T13:52:13.485023847Z 2026-05-03 13:52:13.484 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-03T13:52:13.484384117Z
2026-05-03T13:52:13.487600918Z 2026-05-03 13:52:13.487 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-03T13:52:13.514350188Z 2026-05-03 13:52:13.513 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-03T13:52:13.529164971Z 2026-05-03 13:52:13.528 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-03T13:52:13.529271890Z 2026-05-03 13:52:13.528 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-03T13:52:13.588996663Z 2026-05-03 13:52:13.588 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-03T13:52:14.480203087Z 2026-05-03 13:52:14.479 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-03T13:52:14.767902692Z 2026-05-03 13:52:14.767 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-03T13:52:14.778768019Z 2026-05-03 13:52:14.778 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-03T13:52:14.778801561Z 2026-05-03 13:52:14.778 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-03T13:52:14.783323210Z 2026-05-03 13:52:14.782 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-03T13:52:14.783447404Z 2026-05-03 13:52:14.782 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-03T13:52:14.847303841Z 2026-05-03 13:52:14.846 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-03T13:52:15.285758045Z 2026-05-03 13:52:15.285 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-03T13:52:15.299532800Z 2026-05-03 13:52:15.299 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-03T13:52:15.352443126Z 2026-05-03 13:52:15.351 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-03T13:52:15.387874268Z 2026-05-03 13:52:15.387 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-03T13:52:15.388045698Z 2026-05-03 13:52:15.387 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-03T13:52:15.569205705Z 2026-05-03 13:52:15.568 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-03T13:52:15.593912737Z 2026-05-03 13:52:15.593 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-03T13:52:15.605406837Z 2026-05-03 13:52:15.605 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-03T13:52:15.611228116Z 2026-05-03 13:52:15.610 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72070742595993600-solr-node1:8983_solr-n_0000000000) starting
2026-05-03T13:52:15.710272246Z 2026-05-03 13:52:15.709 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-03T13:52:15.711286767Z 2026-05-03 13:52:15.710 INFO (OverseerStateUpdate-72070742595993600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-03T13:52:15.721749288Z 2026-05-03 13:52:15.721 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-03T13:52:15.780855973Z 2026-05-03 13:52:15.780 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-03T13:52:16.310961537Z 2026-05-03 13:52:16.310 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-03T13:52:16.514555468Z 2026-05-03 13:52:16.514 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-03T13:52:16.893188530Z 2026-05-03 13:52:16.892 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-03T13:52:17.166504822Z 2026-05-03 13:52:17.165 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-03T13:52:17.282118108Z 2026-05-03 13:52:17.281 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_03.request.log
2026-05-03T13:52:17.286153365Z 2026-05-03 13:52:17.285 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-03T13:52:17.287860528Z 2026-05-03 13:52:17.287 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5799ms
2026-05-03T13:52:17.523089750Z 2026-05-03 13:52:17.522 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=57
2026-05-03T13:52:17.728616836Z 2026-05-03 13:52:17.728 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=21
2026-05-03T13:52:21.011221091Z 2026-05-03 13:52:21.010 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-03T13:52:21.167634145Z 2026-05-03 13:52:21.166 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-03T13:52:25.427880196Z 2026-05-03 13:52:25.424 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=17
2026-05-03T13:52:25.455336405Z 2026-05-03 13:52:25.454 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=12
2026-05-03T13:52:25.490041029Z 2026-05-03 13:52:25.489 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=5
2026-05-03T13:52:25.770346812Z 2026-05-03 13:52:25.769 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-03T13:52:25.795381913Z 2026-05-03 13:52:25.794 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-03T13:52:25.802768900Z 2026-05-03 13:52:25.802 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:25.864771243Z 2026-05-03 13:52:25.864 INFO (OverseerStateUpdate-72070742595993600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-03T13:52:25.864798621Z "core":"test-collection_shard3_replica_n1",
2026-05-03T13:52:25.864801891Z "node_name":"solr-node1:8983_solr",
2026-05-03T13:52:25.864804908Z "base_url":"http://solr-node1:8983/solr",
2026-05-03T13:52:25.864807648Z "collection":"test-collection",
2026-05-03T13:52:25.864810417Z "shard":"shard3",
2026-05-03T13:52:25.864813158Z "state":"down",
2026-05-03T13:52:25.864815671Z "type":"NRT",
2026-05-03T13:52:25.864818266Z "operation":"ADDREPLICA",
2026-05-03T13:52:25.864820937Z "waitForFinalState":"false"}
2026-05-03T13:52:25.871653541Z 2026-05-03 13:52:25.871 INFO (OverseerStateUpdate-72070742595993600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-03T13:52:25.871674427Z "core":"test-collection_shard2_replica_n2",
2026-05-03T13:52:25.871677791Z "node_name":"solr-node3:8985_solr",
2026-05-03T13:52:25.871680519Z "base_url":"http://solr-node3:8985/solr",
2026-05-03T13:52:25.871683202Z "collection":"test-collection",
2026-05-03T13:52:25.871685808Z "shard":"shard2",
2026-05-03T13:52:25.871688305Z "state":"down",
2026-05-03T13:52:25.871690810Z "type":"NRT",
2026-05-03T13:52:25.871693427Z "operation":"ADDREPLICA",
2026-05-03T13:52:25.871695978Z "waitForFinalState":"false"}
2026-05-03T13:52:25.875548825Z 2026-05-03 13:52:25.875 INFO (OverseerStateUpdate-72070742595993600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-03T13:52:25.875562710Z "core":"test-collection_shard1_replica_n4",
2026-05-03T13:52:25.875565924Z "node_name":"solr-node2:8984_solr",
2026-05-03T13:52:25.875579031Z "base_url":"http://solr-node2:8984/solr",
2026-05-03T13:52:25.875581281Z "collection":"test-collection",
2026-05-03T13:52:25.875583731Z "shard":"shard1",
2026-05-03T13:52:25.875585651Z "state":"down",
2026-05-03T13:52:25.875587530Z "type":"NRT",
2026-05-03T13:52:25.875589392Z "operation":"ADDREPLICA",
2026-05-03T13:52:25.875591403Z "waitForFinalState":"false"}
2026-05-03T13:52:25.983153335Z 2026-05-03 13:52:25.982 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:26.062196785Z 2026-05-03 13:52:26.061 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-03T13:52:26.157519372Z 2026-05-03 13:52:26.157 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-03T13:52:26.215896481Z 2026-05-03 13:52:26.215 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-03T13:52:26.572817960Z 2026-05-03 13:52:26.572 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-03T13:52:26.572854287Z 2026-05-03 13:52:26.572 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-03T13:52:26.603169661Z 2026-05-03 13:52:26.602 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-03T13:52:27.492504541Z 2026-05-03 13:52:27.492 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-03T13:52:27.524867077Z 2026-05-03 13:52:27.524 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-03T13:52:27.528637240Z 2026-05-03 13:52:27.528 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-03T13:52:27.529531427Z 2026-05-03 13:52:27.529 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-03T13:52:27.529616094Z 2026-05-03 13:52:27.529 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-03T13:52:27.589087981Z 2026-05-03 13:52:27.588 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-03T13:52:27.593485068Z 2026-05-03 13:52:27.593 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-03T13:52:27.595992284Z 2026-05-03 13:52:27.595 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-03T13:52:27.597613987Z 2026-05-03 13:52:27.597 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-03T13:52:27.620059376Z 2026-05-03 13:52:27.619 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-03T13:52:27.629224385Z 2026-05-03 13:52:27.628 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-03T13:52:27.640996185Z 2026-05-03 13:52:27.640 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-03T13:52:27.641034866Z 2026-05-03 13:52:27.640 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-03T13:52:27.649693133Z 2026-05-03 13:52:27.649 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-03T13:52:27.658187668Z 2026-05-03 13:52:27.657 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-03T13:52:27.659900622Z 2026-05-03 13:52:27.659 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-03T13:52:27.677742148Z 2026-05-03 13:52:27.677 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-03T13:52:27.677761835Z 2026-05-03 13:52:27.677 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-03T13:52:27.678031279Z 2026-05-03 13:52:27.677 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-03T13:52:27.678972082Z 2026-05-03 13:52:27.678 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-03T13:52:27.679448985Z 2026-05-03 13:52:27.679 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-03T13:52:27.680520874Z 2026-05-03 13:52:27.680 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72070742595993600-core_node3-n_0000000000
2026-05-03T13:52:27.686369070Z 2026-05-03 13:52:27.686 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-03T13:52:27.792374268Z 2026-05-03 13:52:27.791 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.795290308Z 2026-05-03 13:52:27.795 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-03T13:52:27.803833464Z 2026-05-03 13:52:27.803 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1744
2026-05-03T13:52:27.902433850Z 2026-05-03 13:52:27.901 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1446655,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-03T13:52:10.381755035Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-03T13:52:18.202524712Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-03T13:52:18.202550025Z It should be set to 65000 to avoid operational disruption.
2026-05-03T13:52:18.202553319Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-03T13:52:18.202856213Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-03T13:52:18.202861578Z
2026-05-03T13:52:18.212081277Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-03T13:52:18.219286295Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-03T13:52:18.303192313Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-03T13:52:18.337542849Z WARNING: A command line option has enabled the Security Manager
2026-05-03T13:52:18.337561082Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-03T13:52:19.401972225Z 2026-05-03 13:52:19.398 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-03T13:52:19.686430383Z 2026-05-03 13:52:19.685 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-03T13:52:19.710659767Z 2026-05-03 13:52:19.710 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-03T13:52:19.717287632Z 2026-05-03 13:52:19.716 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-03T13:52:19.717377206Z 2026-05-03 13:52:19.716 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-03T13:52:19.717483720Z 2026-05-03 13:52:19.717 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-03T13:52:19.717598412Z 2026-05-03 13:52:19.717 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-03T13:52:19.717116917Z
2026-05-03T13:52:19.720103344Z 2026-05-03 13:52:19.719 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-03T13:52:19.740748189Z 2026-05-03 13:52:19.740 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-03T13:52:19.763708887Z 2026-05-03 13:52:19.763 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-03T13:52:19.897945084Z 2026-05-03 13:52:19.897 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-03T13:52:19.915058493Z 2026-05-03 13:52:19.914 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-03T13:52:19.915148175Z 2026-05-03 13:52:19.914 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-03T13:52:19.915254593Z 2026-05-03 13:52:19.914 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-03T13:52:20.023872649Z 2026-05-03 13:52:20.023 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-03T13:52:20.023987479Z 2026-05-03 13:52:20.023 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-03T13:52:20.024087942Z 2026-05-03 13:52:20.023 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-03T13:52:20.057685892Z 2026-05-03 13:52:20.057 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-03T13:52:20.907874730Z 2026-05-03 13:52:20.907 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-03T13:52:21.045476870Z 2026-05-03 13:52:21.045 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-03T13:52:21.046953530Z 2026-05-03 13:52:21.045 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-03T13:52:21.055916051Z 2026-05-03 13:52:21.055 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-03T13:52:21.061670101Z 2026-05-03 13:52:21.061 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-03T13:52:21.065581240Z 2026-05-03 13:52:21.065 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-03T13:52:21.065669140Z 2026-05-03 13:52:21.065 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-03T13:52:21.102437268Z 2026-05-03 13:52:21.101 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-03T13:52:21.113340517Z 2026-05-03 13:52:21.112 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-05-03T13:52:21.138064300Z 2026-05-03 13:52:21.137 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-03T13:52:21.150527160Z 2026-05-03 13:52:21.150 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-03T13:52:21.159738405Z 2026-05-03 13:52:21.159 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-03T13:52:21.168182247Z 2026-05-03 13:52:21.167 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-03T13:52:21.208194642Z 2026-05-03 13:52:21.207 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-03T13:52:21.511444424Z 2026-05-03 13:52:21.509 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-03T13:52:21.684888998Z 2026-05-03 13:52:21.684 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-03T13:52:21.964366131Z 2026-05-03 13:52:21.963 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-03T13:52:22.098519119Z 2026-05-03 13:52:22.097 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-03T13:52:22.104205367Z 2026-05-03 13:52:22.103 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_03.request.log
2026-05-03T13:52:22.107716258Z 2026-05-03 13:52:22.107 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-03T13:52:22.108536341Z 2026-05-03 13:52:22.108 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3890ms
2026-05-03T13:52:22.309992076Z 2026-05-03 13:52:22.309 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=43
2026-05-03T13:52:22.357987102Z 2026-05-03 13:52:22.357 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-05-03T13:52:25.731545457Z 2026-05-03 13:52:25.731 INFO (qtp985324122-58-solr-node2-3) [c: s: r: x: t:solr-node2-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=6
2026-05-03T13:52:26.063336357Z 2026-05-03 13:52:26.062 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-03T13:52:26.107374929Z 2026-05-03 13:52:26.106 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-03T13:52:26.163081211Z 2026-05-03 13:52:26.162 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-03T13:52:26.475980084Z 2026-05-03 13:52:26.475 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-03T13:52:26.477056977Z 2026-05-03 13:52:26.475 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-03T13:52:26.495018050Z 2026-05-03 13:52:26.494 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-03T13:52:27.013233431Z 2026-05-03 13:52:27.012 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-03T13:52:27.033272264Z 2026-05-03 13:52:27.032 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-03T13:52:27.038155658Z 2026-05-03 13:52:27.037 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-03T13:52:27.039154504Z 2026-05-03 13:52:27.038 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-03T13:52:27.039228443Z 2026-05-03 13:52:27.039 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-03T13:52:27.102399694Z 2026-05-03 13:52:27.101 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-03T13:52:27.105408154Z 2026-05-03 13:52:27.105 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-03T13:52:27.107490562Z 2026-05-03 13:52:27.107 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-03T13:52:27.109219157Z 2026-05-03 13:52:27.108 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-03T13:52:27.121422383Z 2026-05-03 13:52:27.120 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-03T13:52:27.129710589Z 2026-05-03 13:52:27.129 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-03T13:52:27.136753345Z 2026-05-03 13:52:27.136 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-03T13:52:27.136856307Z 2026-05-03 13:52:27.136 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-03T13:52:27.143745321Z 2026-05-03 13:52:27.143 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-03T13:52:27.156473646Z 2026-05-03 13:52:27.155 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-03T13:52:27.158160056Z 2026-05-03 13:52:27.157 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-03T13:52:27.275284084Z 2026-05-03 13:52:27.274 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-03T13:52:27.275374968Z 2026-05-03 13:52:27.274 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-03T13:52:27.275568730Z 2026-05-03 13:52:27.275 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-03T13:52:27.277132356Z 2026-05-03 13:52:27.276 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-03T13:52:27.277857253Z 2026-05-03 13:52:27.277 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-03T13:52:27.279158914Z 2026-05-03 13:52:27.278 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72070742595993604-core_node6-n_0000000000
2026-05-03T13:52:27.289460538Z 2026-05-03 13:52:27.289 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-03T13:52:27.402387295Z 2026-05-03 13:52:27.401 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.407135729Z 2026-05-03 13:52:27.406 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-03T13:52:27.416969350Z 2026-05-03 13:52:27.416 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1354
2026-05-03T13:52:27.517678539Z 2026-05-03 13:52:27.517 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.792588797Z 2026-05-03 13:52:27.792 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.813110035Z 2026-05-03 13:52:27.812 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-03T13:52:27.902541942Z 2026-05-03 13:52:27.902 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.904991463Z 2026-05-03 13:52:27.904 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2157
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1447071,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-03T13:52:17.844937494Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-03T13:52:18.243921619Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-03T13:52:18.243993015Z It should be set to 65000 to avoid operational disruption.
2026-05-03T13:52:18.243996542Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-03T13:52:18.243998749Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-03T13:52:18.244000816Z
2026-05-03T13:52:18.252319093Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-03T13:52:18.260053812Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-03T13:52:18.303102823Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-03T13:52:18.337532344Z WARNING: A command line option has enabled the Security Manager
2026-05-03T13:52:18.337561195Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-03T13:52:19.388387736Z 2026-05-03 13:52:19.384 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-03T13:52:19.656570919Z 2026-05-03 13:52:19.656 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-03T13:52:19.680932342Z 2026-05-03 13:52:19.680 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-03T13:52:19.688146925Z 2026-05-03 13:52:19.687 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-03T13:52:19.688308478Z 2026-05-03 13:52:19.687 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-03T13:52:19.688333142Z 2026-05-03 13:52:19.687 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-03T13:52:19.688497548Z 2026-05-03 13:52:19.687 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-03T13:52:19.687924056Z
2026-05-03T13:52:19.691011161Z 2026-05-03 13:52:19.690 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-03T13:52:19.714434439Z 2026-05-03 13:52:19.713 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-03T13:52:19.737968845Z 2026-05-03 13:52:19.737 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-03T13:52:19.874606095Z 2026-05-03 13:52:19.874 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-03T13:52:19.892699139Z 2026-05-03 13:52:19.892 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-03T13:52:19.892720939Z 2026-05-03 13:52:19.892 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-03T13:52:19.892724191Z 2026-05-03 13:52:19.892 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-03T13:52:20.002552009Z 2026-05-03 13:52:20.002 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-03T13:52:20.002618602Z 2026-05-03 13:52:20.002 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-03T13:52:20.002708007Z 2026-05-03 13:52:20.002 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-03T13:52:20.035869804Z 2026-05-03 13:52:20.035 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-03T13:52:20.751280298Z 2026-05-03 13:52:20.750 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-03T13:52:20.902407550Z 2026-05-03 13:52:20.901 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-03T13:52:20.904056671Z 2026-05-03 13:52:20.902 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-03T13:52:20.913401434Z 2026-05-03 13:52:20.912 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-03T13:52:20.918927919Z 2026-05-03 13:52:20.918 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-03T13:52:20.922363834Z 2026-05-03 13:52:20.922 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-03T13:52:20.922470347Z 2026-05-03 13:52:20.922 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-03T13:52:20.955679996Z 2026-05-03 13:52:20.955 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-03T13:52:20.963679503Z 2026-05-03 13:52:20.963 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-03T13:52:20.982103121Z 2026-05-03 13:52:20.981 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-03T13:52:20.992248497Z 2026-05-03 13:52:20.991 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-03T13:52:21.001530542Z 2026-05-03 13:52:21.001 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-03T13:52:21.012196739Z 2026-05-03 13:52:21.011 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-03T13:52:21.051670624Z 2026-05-03 13:52:21.051 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-03T13:52:21.167665442Z 2026-05-03 13:52:21.167 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-03T13:52:21.329445684Z 2026-05-03 13:52:21.328 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-03T13:52:21.494856142Z 2026-05-03 13:52:21.494 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-03T13:52:21.755374618Z 2026-05-03 13:52:21.754 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-03T13:52:21.885077165Z 2026-05-03 13:52:21.884 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-03T13:52:21.890503548Z 2026-05-03 13:52:21.890 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_03.request.log
2026-05-03T13:52:21.894017366Z 2026-05-03 13:52:21.893 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-03T13:52:21.894958701Z 2026-05-03 13:52:21.894 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3636ms
2026-05-03T13:52:24.099669357Z 2026-05-03 13:52:24.093 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=73
2026-05-03T13:52:24.171103383Z 2026-05-03 13:52:24.170 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19
2026-05-03T13:52:26.064917863Z 2026-05-03 13:52:26.063 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-03T13:52:26.123030267Z 2026-05-03 13:52:26.122 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-03T13:52:26.195025507Z 2026-05-03 13:52:26.194 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-03T13:52:26.540107740Z 2026-05-03 13:52:26.539 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-03T13:52:26.540134036Z 2026-05-03 13:52:26.539 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-03T13:52:26.564872936Z 2026-05-03 13:52:26.564 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-03T13:52:27.014002299Z 2026-05-03 13:52:27.013 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-03T13:52:27.035002075Z 2026-05-03 13:52:27.034 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-03T13:52:27.041566519Z 2026-05-03 13:52:27.041 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-03T13:52:27.042913881Z 2026-05-03 13:52:27.042 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-03T13:52:27.042925775Z 2026-05-03 13:52:27.042 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-03T13:52:27.105947262Z 2026-05-03 13:52:27.105 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-03T13:52:27.109139320Z 2026-05-03 13:52:27.108 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-03T13:52:27.111658750Z 2026-05-03 13:52:27.111 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-03T13:52:27.113536236Z 2026-05-03 13:52:27.113 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-03T13:52:27.125326888Z 2026-05-03 13:52:27.124 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-03T13:52:27.133363472Z 2026-05-03 13:52:27.132 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-03T13:52:27.139990701Z 2026-05-03 13:52:27.139 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-03T13:52:27.140022001Z 2026-05-03 13:52:27.139 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-03T13:52:27.148031009Z 2026-05-03 13:52:27.147 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-03T13:52:27.160540339Z 2026-05-03 13:52:27.159 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-03T13:52:27.161965643Z 2026-05-03 13:52:27.161 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-03T13:52:27.280534913Z 2026-05-03 13:52:27.279 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-03T13:52:27.280551337Z 2026-05-03 13:52:27.280 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-03T13:52:27.280554379Z 2026-05-03 13:52:27.280 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-03T13:52:27.284997855Z 2026-05-03 13:52:27.284 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-03T13:52:27.285688761Z 2026-05-03 13:52:27.285 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-03T13:52:27.287180353Z 2026-05-03 13:52:27.286 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72070742595993603-core_node5-n_0000000000
2026-05-03T13:52:27.297711243Z 2026-05-03 13:52:27.297 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-03T13:52:27.402683444Z 2026-05-03 13:52:27.402 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.407052529Z 2026-05-03 13:52:27.406 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-03T13:52:27.417083282Z 2026-05-03 13:52:27.416 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1354
2026-05-03T13:52:27.517264355Z 2026-05-03 13:52:27.516 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.792592769Z 2026-05-03 13:52:27.792 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-03T13:52:27.902574766Z 2026-05-03 13:52:27.902 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1447199,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-03T13:52:18.044727322Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 23742 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# f7393cba8d0f: Pulling fs layer
# 6eaaa8b64b5f: Pulling fs layer
# e3a891d79218: Pulling fs layer
# 3aa5cf732722: Pulling fs layer
# 9f9f08cf5abb: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 3aa5cf732722: Waiting
# 9f9f08cf5abb: Waiting
# 4f4fb700ef54: Waiting
# e3a891d79218: Verifying Checksum
# e3a891d79218: Download complete
# 6eaaa8b64b5f: Verifying Checksum
# 6eaaa8b64b5f: Download complete
# 3aa5cf732722: Verifying Checksum
# 3aa5cf732722: Download complete
# 9f9f08cf5abb: Verifying Checksum
# 9f9f08cf5abb: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# f7393cba8d0f: Verifying Checksum
# f7393cba8d0f: Download complete
# f7393cba8d0f: Pull complete
# 6eaaa8b64b5f: Pull complete
# e3a891d79218: Pull complete
# 3aa5cf732722: Pull complete
# 9f9f08cf5abb: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:acdab01f12ef3790dec34d2ef3f49eac433a7532294d74102f44dbeba80e42ad
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# d0a3119775940a765bd5280dec49b8a90ffc49d898d9b1075bf36a10ded5ae96
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 140134aae0cc292894c61d783770950c9c94525379c9bc4a4edba38da21f110d
# Starting solr-node2 connected to first node's ZooKeeper
# 56296288ecfbd0041d1ada086fca378a4f0c46770f2ad19ee9595da6d1d45a76
# Started solr-node3 connected to first node's ZooKeeper
# 9ff5cb6a1bd5a03b0acb910c4222ae2d2681da92afdcad178e762ae47ce36813
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-03 13:52:25.522; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-03 13:52:25.562; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node2:8984/solr] to [http://solr-node2:8984/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14937 ms
ok 85 snapshot lifecycle # in 4690 ms
ok 86 snapshot list # in 3279 ms
ok 87 snapshot describe # in 3352 ms
ok 88 start solr with ssl # in 16055 ms
ok 89 use different hostname when not checking peer-name # in 24193 ms
ok 90 start solr with ssl and auth # in 14589 ms
ok 91 start solr with client truststore and security manager # in 16215 ms
ok 92 start solr with mTLS needed # in 46896 ms
ok 93 start solr with mTLS wanted # in 34129 ms
ok 94 test keystore reload # in 59018 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15779 ms
ok 96 stop command for single port # in 18264 ms
ok 97 check stop command doesn't hang # in 223 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8170 ms
ok 99 deprecated system properties converted to modern properties # in 195089 ms
ok 100 start with custom jetty options # in 8056 ms
ok 101 bootstrapping a configset # in 9162 ms
ok 102 status detects locally running solr # in 9469 ms
ok 103 status with --solr-url from user # in 8089 ms
ok 104 status with --port from user # in 7957 ms
ok 105 multiple connection options are prevented # in 852 ms
ok 106 status with invalid --solr-url from user # in 1307 ms
ok 107 status with --short format # in 195910 ms
ok 108 searching solr via locally executed streaming expression # in 1751 ms
ok 109 searching solr via remotely executed streaming expression # in 1268 ms
ok 110 variable interpolation # in 1690 ms
ok 111 searching solr without credentials fails with error # in 1081 ms
ok 112 --version returns Solr version # in 721 ms
ok 113 version as direct tool call still runs # in 719 ms
ok 114 short help # in 790 ms
ok 115 short help is inferred # in 779 ms
ok 116 long help # in 798 ms
ok 117 running subcommands with zk is prevented # in 721 ms
ok 118 listing out files # in 2135 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7760 ms
ok 120 copying files around # in 10957 ms
ok 121 upconfig # in 2530 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1197 ms
ok 123 downconfig # in 1185 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1437 ms
ok 125 env var ZK_HOST is honored # in 3220 ms
ok 126 Cleanup # in 1846 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 30s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/zncmn3iesdhsc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 197 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/197/
No tests ran.
Build Log:
[...truncated 424 lines...]
# ERROR: Failed to create collection 'test_args' due to: IOException occurred when talking to server at: http://localhost:43553/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #10 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/79-10
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43553.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43553.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:43553/solr/test_args/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_args -s http://localhost:43553 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-05-02 13:08:35.332; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-02 13:08:35.336; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_args' due to: IOException occurred when talking to server at: http://localhost:43553/solr/admin/collections
not ok 80 stdin mode # in 1087 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 201)
# `assert_output --partial "Created collection 'test_stdin'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'test_stdin'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43553.
#
# ERROR: Server refused connection at: http://localhost:43553/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43553.
#
# ERROR: Server refused connection at: http://localhost:43553/solr/admin/info/system
not ok 81 verbose echo the Solr response # in 1070 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43553.
#
# ERROR: Server refused connection at: http://localhost:43553/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43553.
#
# ERROR: Server refused connection at: http://localhost:43553/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8624 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 21 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-05-02T13:09:02.189002076Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-02T13:09:02.189058195Z It should be set to 65000 to avoid operational disruption.
2026-05-02T13:09:02.189061369Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-02T13:09:02.192057672Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-02T13:09:02.192073764Z
2026-05-02T13:09:02.207840826Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-02T13:09:02.219557722Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-02T13:09:02.271995933Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-02T13:09:02.412212331Z WARNING: A command line option has enabled the Security Manager
2026-05-02T13:09:02.412239418Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-02T13:09:04.184153941Z 2026-05-02 13:09:04.181 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-02T13:09:04.453655804Z 2026-05-02 13:09:04.452 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-02T13:09:04.565838478Z 2026-05-02 13:09:04.565 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-02T13:09:04.590401653Z 2026-05-02 13:09:04.589 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-02T13:09:04.590494525Z 2026-05-02 13:09:04.590 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-05-02T13:09:04.590631125Z 2026-05-02 13:09:04.590 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-02T13:09:04.590741176Z 2026-05-02 13:09:04.590 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-02T13:09:04.590216774Z
2026-05-02T13:09:04.593346150Z 2026-05-02 13:09:04.592 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-02T13:09:04.617162618Z 2026-05-02 13:09:04.616 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-02T13:09:04.679238488Z 2026-05-02 13:09:04.678 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-02T13:09:04.679258372Z 2026-05-02 13:09:04.678 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-02T13:09:04.758475732Z 2026-05-02 13:09:04.757 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-02T13:09:05.807529951Z 2026-05-02 13:09:05.806 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-02T13:09:06.246027839Z 2026-05-02 13:09:06.245 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-02T13:09:06.267349622Z 2026-05-02 13:09:06.266 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-05-02T13:09:06.267441620Z 2026-05-02 13:09:06.266 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-05-02T13:09:06.302826530Z 2026-05-02 13:09:06.302 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-05-02T13:09:06.302922149Z 2026-05-02 13:09:06.302 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-05-02T13:09:06.393166319Z 2026-05-02 13:09:06.392 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-05-02T13:09:06.805931053Z 2026-05-02 13:09:06.805 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-05-02T13:09:06.852053615Z 2026-05-02 13:09:06.851 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-02T13:09:07.092381665Z 2026-05-02 13:09:07.091 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-02T13:09:07.141442463Z 2026-05-02 13:09:07.139 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-02T13:09:07.141647623Z 2026-05-02 13:09:07.139 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-02T13:09:07.298142431Z 2026-05-02 13:09:07.297 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-02T13:09:07.399129597Z 2026-05-02 13:09:07.398 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-02T13:09:07.414380257Z 2026-05-02 13:09:07.413 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-05-02T13:09:07.473404889Z 2026-05-02 13:09:07.472 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72064824924569600-solr-node1:8983_solr-n_0000000000) starting
2026-05-02T13:09:07.596552361Z 2026-05-02 13:09:07.595 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-05-02T13:09:07.597502949Z 2026-05-02 13:09:07.596 INFO (OverseerStateUpdate-72064824924569600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-05-02T13:09:07.613435376Z 2026-05-02 13:09:07.612 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-02T13:09:07.680758696Z 2026-05-02 13:09:07.679 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-02T13:09:08.573712943Z 2026-05-02 13:09:08.572 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-02T13:09:08.822331622Z 2026-05-02 13:09:08.821 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-02T13:09:09.164178791Z 2026-05-02 13:09:09.163 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-02T13:09:09.463698599Z 2026-05-02 13:09:09.461 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-02T13:09:09.471627460Z 2026-05-02 13:09:09.471 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_02.request.log
2026-05-02T13:09:09.476712498Z 2026-05-02 13:09:09.476 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-05-02T13:09:09.477793320Z 2026-05-02 13:09:09.477 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @7259ms
2026-05-02T13:09:10.229026813Z 2026-05-02 13:09:10.228 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=75
2026-05-02T13:09:10.471005410Z 2026-05-02 13:09:10.470 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=35
2026-05-02T13:09:13.947049086Z 2026-05-02 13:09:13.946 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-02T13:09:13.966438252Z 2026-05-02 13:09:13.965 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-02T13:09:19.016916612Z 2026-05-02 13:09:19.011 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=14
2026-05-02T13:09:19.073482435Z 2026-05-02 13:09:19.073 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11
2026-05-02T13:09:19.105436123Z 2026-05-02 13:09:19.105 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=6
2026-05-02T13:09:19.532519188Z 2026-05-02 13:09:19.532 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-05-02T13:09:19.566449557Z 2026-05-02 13:09:19.566 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-05-02T13:09:19.583510419Z 2026-05-02 13:09:19.582 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:19.706979634Z 2026-05-02 13:09:19.706 INFO (OverseerStateUpdate-72064824924569600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-02T13:09:19.707000393Z "core":"test-collection_shard3_replica_n1",
2026-05-02T13:09:19.707003996Z "node_name":"solr-node1:8983_solr",
2026-05-02T13:09:19.707006744Z "base_url":"http://solr-node1:8983/solr",
2026-05-02T13:09:19.707009358Z "collection":"test-collection",
2026-05-02T13:09:19.707011998Z "shard":"shard3",
2026-05-02T13:09:19.707014806Z "state":"down",
2026-05-02T13:09:19.707017252Z "type":"NRT",
2026-05-02T13:09:19.707019647Z "operation":"ADDREPLICA",
2026-05-02T13:09:19.707022085Z "waitForFinalState":"false"}
2026-05-02T13:09:19.719971831Z 2026-05-02 13:09:19.719 INFO (OverseerStateUpdate-72064824924569600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-02T13:09:19.719990893Z "core":"test-collection_shard2_replica_n2",
2026-05-02T13:09:19.719994268Z "node_name":"solr-node3:8985_solr",
2026-05-02T13:09:19.719996899Z "base_url":"http://solr-node3:8985/solr",
2026-05-02T13:09:19.719999591Z "collection":"test-collection",
2026-05-02T13:09:19.720002313Z "shard":"shard2",
2026-05-02T13:09:19.720004806Z "state":"down",
2026-05-02T13:09:19.720007285Z "type":"NRT",
2026-05-02T13:09:19.720009862Z "operation":"ADDREPLICA",
2026-05-02T13:09:19.720012592Z "waitForFinalState":"false"}
2026-05-02T13:09:19.724479839Z 2026-05-02 13:09:19.723 INFO (OverseerStateUpdate-72064824924569600-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-05-02T13:09:19.724496829Z "core":"test-collection_shard1_replica_n4",
2026-05-02T13:09:19.724500425Z "node_name":"solr-node2:8984_solr",
2026-05-02T13:09:19.724513692Z "base_url":"http://solr-node2:8984/solr",
2026-05-02T13:09:19.724515832Z "collection":"test-collection",
2026-05-02T13:09:19.724517785Z "shard":"shard1",
2026-05-02T13:09:19.724519665Z "state":"down",
2026-05-02T13:09:19.724521625Z "type":"NRT",
2026-05-02T13:09:19.724523465Z "operation":"ADDREPLICA",
2026-05-02T13:09:19.724525593Z "waitForFinalState":"false"}
2026-05-02T13:09:19.833813404Z 2026-05-02 13:09:19.833 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:19.992797951Z 2026-05-02 13:09:19.992 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-05-02T13:09:20.057062242Z 2026-05-02 13:09:20.056 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-02T13:09:20.128085525Z 2026-05-02 13:09:20.127 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-02T13:09:20.525453300Z 2026-05-02 13:09:20.524 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-02T13:09:20.525514714Z 2026-05-02 13:09:20.525 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-05-02T13:09:20.547625089Z 2026-05-02 13:09:20.547 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-05-02T13:09:21.857319533Z 2026-05-02 13:09:21.855 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-02T13:09:21.969161303Z 2026-05-02 13:09:21.968 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-02T13:09:21.975682526Z 2026-05-02 13:09:21.974 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-02T13:09:21.976429017Z 2026-05-02 13:09:21.975 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-02T13:09:21.976535566Z 2026-05-02 13:09:21.976 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-02T13:09:22.074373465Z 2026-05-02 13:09:22.073 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-02T13:09:22.080263005Z 2026-05-02 13:09:22.079 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-02T13:09:22.082484239Z 2026-05-02 13:09:22.082 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-02T13:09:22.084521689Z 2026-05-02 13:09:22.084 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-02T13:09:22.101572730Z 2026-05-02 13:09:22.101 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-05-02T13:09:22.113488820Z 2026-05-02 13:09:22.113 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-02T13:09:22.124426109Z 2026-05-02 13:09:22.124 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-02T13:09:22.124476009Z 2026-05-02 13:09:22.124 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-02T13:09:22.137077094Z 2026-05-02 13:09:22.136 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-02T13:09:22.141271663Z 2026-05-02 13:09:22.140 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-05-02T13:09:22.142733277Z 2026-05-02 13:09:22.142 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-05-02T13:09:22.156921435Z 2026-05-02 13:09:22.153 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-02T13:09:22.156936501Z 2026-05-02 13:09:22.156 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-02T13:09:22.156997617Z 2026-05-02 13:09:22.156 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-05-02T13:09:22.158062620Z 2026-05-02 13:09:22.157 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-02T13:09:22.158473709Z 2026-05-02 13:09:22.158 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-05-02T13:09:22.159347681Z 2026-05-02 13:09:22.159 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72064824924569600-core_node3-n_0000000000
2026-05-02T13:09:22.164031239Z 2026-05-02 13:09:22.163 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-05-02T13:09:22.269136247Z 2026-05-02 13:09:22.268 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:22.271248689Z 2026-05-02 13:09:22.271 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-02T13:09:22.279792847Z 2026-05-02 13:09:22.279 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2304
2026-05-02T13:09:22.377972509Z 2026-05-02 13:09:22.377 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1044648,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-02T13:09:00.816176148Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-05-02T13:09:11.083484986Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-02T13:09:11.083512956Z It should be set to 65000 to avoid operational disruption.
2026-05-02T13:09:11.083516302Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-02T13:09:11.084681458Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-02T13:09:11.084688010Z
2026-05-02T13:09:11.093454295Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-02T13:09:11.102635303Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-02T13:09:11.149384988Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-02T13:09:11.186838440Z WARNING: A command line option has enabled the Security Manager
2026-05-02T13:09:11.186856083Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-02T13:09:12.247771712Z 2026-05-02 13:09:12.244 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-02T13:09:12.511846499Z 2026-05-02 13:09:12.511 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-02T13:09:12.539522894Z 2026-05-02 13:09:12.538 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-02T13:09:12.546050579Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-02T13:09:12.546150252Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-05-02T13:09:12.546335386Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-02T13:09:12.546382584Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-02T13:09:12.545881958Z
2026-05-02T13:09:12.548880552Z 2026-05-02 13:09:12.548 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-02T13:09:12.571425441Z 2026-05-02 13:09:12.570 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-02T13:09:12.597179699Z 2026-05-02 13:09:12.596 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-02T13:09:12.736550457Z 2026-05-02 13:09:12.736 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-02T13:09:12.754200451Z 2026-05-02 13:09:12.753 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-02T13:09:12.754219654Z 2026-05-02 13:09:12.753 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-02T13:09:12.754330435Z 2026-05-02 13:09:12.754 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-02T13:09:12.864517301Z 2026-05-02 13:09:12.863 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-02T13:09:12.864598974Z 2026-05-02 13:09:12.864 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-02T13:09:12.864682029Z 2026-05-02 13:09:12.864 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-02T13:09:12.897724822Z 2026-05-02 13:09:12.897 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-02T13:09:13.666451125Z 2026-05-02 13:09:13.665 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-02T13:09:13.831816888Z 2026-05-02 13:09:13.831 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-02T13:09:13.833030957Z 2026-05-02 13:09:13.831 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-02T13:09:13.841185676Z 2026-05-02 13:09:13.840 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-02T13:09:13.847608007Z 2026-05-02 13:09:13.847 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-02T13:09:13.854555723Z 2026-05-02 13:09:13.851 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-02T13:09:13.854660127Z 2026-05-02 13:09:13.852 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-02T13:09:13.887327096Z 2026-05-02 13:09:13.886 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-02T13:09:13.895626139Z 2026-05-02 13:09:13.895 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-02T13:09:13.916034458Z 2026-05-02 13:09:13.915 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-02T13:09:13.927651916Z 2026-05-02 13:09:13.927 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-05-02T13:09:13.939966802Z 2026-05-02 13:09:13.939 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-05-02T13:09:13.948228090Z 2026-05-02 13:09:13.947 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-02T13:09:13.966969225Z 2026-05-02 13:09:13.966 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-02T13:09:13.990633388Z 2026-05-02 13:09:13.990 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-02T13:09:14.288685032Z 2026-05-02 13:09:14.287 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-02T13:09:14.465141944Z 2026-05-02 13:09:14.464 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-02T13:09:14.724734989Z 2026-05-02 13:09:14.724 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-02T13:09:14.867050612Z 2026-05-02 13:09:14.866 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-02T13:09:14.872209917Z 2026-05-02 13:09:14.871 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_02.request.log
2026-05-02T13:09:14.875822371Z 2026-05-02 13:09:14.875 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-05-02T13:09:14.877139069Z 2026-05-02 13:09:14.876 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3777ms
2026-05-02T13:09:15.063276204Z 2026-05-02 13:09:15.062 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41
2026-05-02T13:09:15.119452930Z 2026-05-02 13:09:15.118 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-05-02T13:09:19.976763336Z 2026-05-02 13:09:19.976 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-05-02T13:09:20.028678262Z 2026-05-02 13:09:20.028 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-02T13:09:20.125353615Z 2026-05-02 13:09:20.124 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-02T13:09:20.451748248Z 2026-05-02 13:09:20.451 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-02T13:09:20.451813956Z 2026-05-02 13:09:20.451 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-05-02T13:09:20.474446043Z 2026-05-02 13:09:20.473 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-05-02T13:09:21.182547836Z 2026-05-02 13:09:21.181 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-02T13:09:21.203627773Z 2026-05-02 13:09:21.203 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-02T13:09:21.209477155Z 2026-05-02 13:09:21.209 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-02T13:09:21.210577829Z 2026-05-02 13:09:21.209 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-02T13:09:21.210685721Z 2026-05-02 13:09:21.210 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-02T13:09:21.324418523Z 2026-05-02 13:09:21.324 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-02T13:09:21.328024686Z 2026-05-02 13:09:21.327 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-02T13:09:21.330399012Z 2026-05-02 13:09:21.330 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-02T13:09:21.334589093Z 2026-05-02 13:09:21.334 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-02T13:09:21.349757135Z 2026-05-02 13:09:21.349 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-05-02T13:09:21.357568114Z 2026-05-02 13:09:21.357 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-02T13:09:21.364473188Z 2026-05-02 13:09:21.364 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-02T13:09:21.364533996Z 2026-05-02 13:09:21.364 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-02T13:09:21.372286893Z 2026-05-02 13:09:21.371 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-02T13:09:21.384193385Z 2026-05-02 13:09:21.383 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-05-02T13:09:21.385620051Z 2026-05-02 13:09:21.385 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-05-02T13:09:21.403855799Z 2026-05-02 13:09:21.403 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-02T13:09:21.403998052Z 2026-05-02 13:09:21.403 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-02T13:09:21.404007754Z 2026-05-02 13:09:21.403 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-05-02T13:09:21.405540617Z 2026-05-02 13:09:21.405 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-02T13:09:21.406373501Z 2026-05-02 13:09:21.406 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-05-02T13:09:21.407408105Z 2026-05-02 13:09:21.407 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72064824924569603-core_node6-n_0000000000
2026-05-02T13:09:21.429467205Z 2026-05-02 13:09:21.429 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-05-02T13:09:21.537137907Z 2026-05-02 13:09:21.536 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:21.542121963Z 2026-05-02 13:09:21.541 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-02T13:09:21.550921666Z 2026-05-02 13:09:21.550 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1575
2026-05-02T13:09:21.656239532Z 2026-05-02 13:09:21.655 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:22.269477244Z 2026-05-02 13:09:22.269 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:22.378142928Z 2026-05-02 13:09:22.377 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1045049,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-02T13:09:10.702013865Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-05-02T13:09:11.137965647Z *** [WARN] *** Your open file limit is currently 1024.
2026-05-02T13:09:11.137994803Z It should be set to 65000 to avoid operational disruption.
2026-05-02T13:09:11.137997936Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-05-02T13:09:11.138000069Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-05-02T13:09:11.138002189Z
2026-05-02T13:09:11.146618876Z Java 17 detected. Enabled workaround for SOLR-16463
2026-05-02T13:09:11.154171043Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-05-02T13:09:11.197275687Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-05-02T13:09:11.227420029Z WARNING: A command line option has enabled the Security Manager
2026-05-02T13:09:11.227441507Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-05-02T13:09:12.247585541Z 2026-05-02 13:09:12.244 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-05-02T13:09:12.512063774Z 2026-05-02 13:09:12.511 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-05-02T13:09:12.539558057Z 2026-05-02 13:09:12.539 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-05-02T13:09:12.546077227Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-05-02T13:09:12.546327479Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-05-02T13:09:12.546361423Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-05-02T13:09:12.546535994Z 2026-05-02 13:09:12.545 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-02T13:09:12.545949893Z
2026-05-02T13:09:12.548981770Z 2026-05-02 13:09:12.548 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-05-02T13:09:12.571418384Z 2026-05-02 13:09:12.570 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-05-02T13:09:12.597144460Z 2026-05-02 13:09:12.596 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-02T13:09:12.736506141Z 2026-05-02 13:09:12.736 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-05-02T13:09:12.754702991Z 2026-05-02 13:09:12.754 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-02T13:09:12.754842551Z 2026-05-02 13:09:12.754 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-02T13:09:12.754976754Z 2026-05-02 13:09:12.754 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-05-02T13:09:12.864454065Z 2026-05-02 13:09:12.863 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-05-02T13:09:12.864499134Z 2026-05-02 13:09:12.864 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-05-02T13:09:12.864629558Z 2026-05-02 13:09:12.864 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-05-02T13:09:12.898357783Z 2026-05-02 13:09:12.897 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-05-02T13:09:13.726392315Z 2026-05-02 13:09:13.725 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-05-02T13:09:13.865449253Z 2026-05-02 13:09:13.864 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-05-02T13:09:13.866675128Z 2026-05-02 13:09:13.864 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-05-02T13:09:13.874560834Z 2026-05-02 13:09:13.874 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-02T13:09:13.880032227Z 2026-05-02 13:09:13.879 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-05-02T13:09:13.882229999Z 2026-05-02 13:09:13.881 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-05-02T13:09:13.882338473Z 2026-05-02 13:09:13.882 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-05-02T13:09:13.909654058Z 2026-05-02 13:09:13.909 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-05-02T13:09:13.917220825Z 2026-05-02 13:09:13.916 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-05-02T13:09:13.936433448Z 2026-05-02 13:09:13.936 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-05-02T13:09:13.947651110Z 2026-05-02 13:09:13.947 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-05-02T13:09:13.949364749Z 2026-05-02 13:09:13.949 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-05-02T13:09:13.958977083Z 2026-05-02 13:09:13.958 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-05-02T13:09:13.966993505Z 2026-05-02 13:09:13.966 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-05-02T13:09:14.015281588Z 2026-05-02 13:09:14.014 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-05-02T13:09:14.283321712Z 2026-05-02 13:09:14.282 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-05-02T13:09:14.463674950Z 2026-05-02 13:09:14.463 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-05-02T13:09:14.720594972Z 2026-05-02 13:09:14.720 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-02T13:09:14.856859941Z 2026-05-02 13:09:14.856 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-05-02T13:09:14.862312110Z 2026-05-02 13:09:14.861 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_02.request.log
2026-05-02T13:09:14.866260869Z 2026-05-02 13:09:14.865 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-05-02T13:09:14.867058588Z 2026-05-02 13:09:14.866 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3714ms
2026-05-02T13:09:17.266079614Z 2026-05-02 13:09:17.256 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=91
2026-05-02T13:09:17.339254531Z 2026-05-02 13:09:17.338 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-05-02T13:09:19.467344690Z 2026-05-02 13:09:19.466 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=20
2026-05-02T13:09:19.988075085Z 2026-05-02 13:09:19.987 INFO (qtp985324122-30-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-05-02T13:09:20.049727181Z 2026-05-02 13:09:20.049 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-05-02T13:09:20.154497126Z 2026-05-02 13:09:20.152 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-05-02T13:09:20.576819166Z 2026-05-02 13:09:20.576 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-05-02T13:09:20.576907396Z 2026-05-02 13:09:20.576 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-05-02T13:09:20.595852254Z 2026-05-02 13:09:20.595 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-05-02T13:09:21.189554974Z 2026-05-02 13:09:21.188 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-05-02T13:09:21.210894478Z 2026-05-02 13:09:21.210 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-05-02T13:09:21.219834685Z 2026-05-02 13:09:21.219 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-05-02T13:09:21.220920975Z 2026-05-02 13:09:21.220 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-05-02T13:09:21.220984612Z 2026-05-02 13:09:21.220 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-05-02T13:09:21.323738190Z 2026-05-02 13:09:21.323 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-05-02T13:09:21.329079792Z 2026-05-02 13:09:21.327 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-05-02T13:09:21.331421304Z 2026-05-02 13:09:21.331 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-05-02T13:09:21.334410787Z 2026-05-02 13:09:21.334 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-05-02T13:09:21.349754916Z 2026-05-02 13:09:21.349 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-05-02T13:09:21.357521901Z 2026-05-02 13:09:21.357 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-05-02T13:09:21.364415693Z 2026-05-02 13:09:21.364 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-05-02T13:09:21.364532752Z 2026-05-02 13:09:21.364 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-05-02T13:09:21.370709410Z 2026-05-02 13:09:21.370 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-05-02T13:09:21.382973142Z 2026-05-02 13:09:21.382 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-05-02T13:09:21.384341924Z 2026-05-02 13:09:21.384 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-05-02T13:09:21.403022362Z 2026-05-02 13:09:21.402 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-05-02T13:09:21.403104756Z 2026-05-02 13:09:21.402 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-05-02T13:09:21.403118599Z 2026-05-02 13:09:21.402 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-05-02T13:09:21.404921942Z 2026-05-02 13:09:21.404 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-05-02T13:09:21.405906531Z 2026-05-02 13:09:21.405 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-05-02T13:09:21.406976714Z 2026-05-02 13:09:21.406 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72064824924569604-core_node5-n_0000000000
2026-05-02T13:09:21.424966951Z 2026-05-02 13:09:21.424 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-05-02T13:09:21.536986908Z 2026-05-02 13:09:21.536 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:21.542796234Z 2026-05-02 13:09:21.542 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-05-02T13:09:21.552114083Z 2026-05-02 13:09:21.551 INFO (qtp985324122-30-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1577
2026-05-02T13:09:21.656185905Z 2026-05-02 13:09:21.655 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:22.269359207Z 2026-05-02 13:09:22.268 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:22.290654360Z 2026-05-02 13:09:22.290 INFO (qtp985324122-27-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-05-02T13:09:22.378106301Z 2026-05-02 13:09:22.377 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-05-02T13:09:22.380244190Z 2026-05-02 13:09:22.379 INFO (qtp985324122-27-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2893
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1045177,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-05-02T13:09:10.934065518Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 27479 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# 664a8913b942: Pulling fs layer
# 365f54d73ad0: Pulling fs layer
# e51c1130faba: Pulling fs layer
# c1916897f8d4: Pulling fs layer
# 4a7802b482e8: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 4a7802b482e8: Waiting
# 4f4fb700ef54: Waiting
# c1916897f8d4: Waiting
# 365f54d73ad0: Verifying Checksum
# 365f54d73ad0: Download complete
# e51c1130faba: Verifying Checksum
# e51c1130faba: Download complete
# c1916897f8d4: Verifying Checksum
# c1916897f8d4: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 4a7802b482e8: Verifying Checksum
# 4a7802b482e8: Download complete
# 664a8913b942: Verifying Checksum
# 664a8913b942: Download complete
# 664a8913b942: Pull complete
# 365f54d73ad0: Pull complete
# e51c1130faba: Pull complete
# c1916897f8d4: Pull complete
# 4a7802b482e8: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:1b9b047de82f767de67c4530c4badf11df9b82863686c7460cadaa3923634136
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 975b6482028a8dbc55376f14593959badba9d5c9d2449227cfa941dd0c186585
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 386cc8cb38256d89c03237cdce1964174c5e7a6763dfc7d8dbdea97199a94547
# Starting solr-node2 connected to first node's ZooKeeper
# 1436168b1607c50fac3715d652899b789e18b1b6bc1f71d546e52df80b428107
# Started solr-node3 connected to first node's ZooKeeper
# 5c1335d8c4c922e1442f5fa57843cbc32161119a41367e2e4549253fa25db988
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-05-02 13:09:19.149; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-05-02 13:09:19.207; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14501 ms
ok 85 snapshot lifecycle # in 4512 ms
ok 86 snapshot list # in 3099 ms
ok 87 snapshot describe # in 3131 ms
ok 88 start solr with ssl # in 15480 ms
ok 89 use different hostname when not checking peer-name # in 23425 ms
ok 90 start solr with ssl and auth # in 13830 ms
ok 91 start solr with client truststore and security manager # in 15897 ms
ok 92 start solr with mTLS needed # in 43922 ms
ok 93 start solr with mTLS wanted # in 32512 ms
ok 94 test keystore reload # in 56190 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15512 ms
ok 96 stop command for single port # in 17671 ms
ok 97 check stop command doesn't hang # in 203 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7714 ms
ok 99 deprecated system properties converted to modern properties # in 195176 ms
ok 100 start with custom jetty options # in 7809 ms
ok 101 bootstrapping a configset # in 9106 ms
ok 102 status detects locally running solr # in 9097 ms
ok 103 status with --solr-url from user # in 7728 ms
ok 104 status with --port from user # in 7734 ms
ok 105 multiple connection options are prevented # in 802 ms
ok 106 status with invalid --solr-url from user # in 1130 ms
ok 107 status with --short format # in 196012 ms
ok 108 searching solr via locally executed streaming expression # in 1622 ms
ok 109 searching solr via remotely executed streaming expression # in 1209 ms
ok 110 variable interpolation # in 1604 ms
ok 111 searching solr without credentials fails with error # in 1111 ms
ok 112 --version returns Solr version # in 720 ms
ok 113 version as direct tool call still runs # in 696 ms
ok 114 short help # in 726 ms
ok 115 short help is inferred # in 741 ms
ok 116 long help # in 738 ms
ok 117 running subcommands with zk is prevented # in 682 ms
ok 118 listing out files # in 2079 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7431 ms
ok 120 copying files around # in 10527 ms
ok 121 upconfig # in 2531 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1077 ms
ok 123 downconfig # in 1110 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1359 ms
ok 125 env var ZK_HOST is honored # in 3086 ms
ok 126 Cleanup # in 1804 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 18s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/dkmfivpci7mks
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 196 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/196/
No tests ran.
Build Log:
[...truncated 254 lines...]
BUILD FAILED in 3m 47s
78 actionable tasks: 78 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/cttalr6e5rscc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 195 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/195/
No tests ran.
Build Log:
[...truncated 254 lines...]
BUILD FAILED in 3m 55s
78 actionable tasks: 78 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/damh2lswmnrx6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 194 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/194/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-04-29 12:51:35.083; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-29 12:51:35.087; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-29 12:51:35.586; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-29 12:51:35.586; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 17393 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 45205 is already being used by another process (pid: 2232936)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-04-29 12:51:36.891; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-29 12:51:36.895; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 10 post, postlogs and export with basic auth # in 18904 ms
ok 11 package with basic auth # in 12823 ms
ok 12 admin UI static content is excepted from basic auth # in 11040 ms
ok 13 nothing # in 11 ms
ok 14 server does not compress response without Accept-Encoding header # in 152 ms
ok 15 server compresses response when Accept-Encoding: gzip is requested # in 88 ms
ok 16 compressed response can be decompressed and parsed # in 85 ms
ok 17 setting property # in 11565 ms
ok 18 short form of setting property # in 6700 ms
ok 19 connecting to solr via various solr urls and zk hosts # in 9891 ms
ok 20 create for non cloud mode # in 8120 ms
ok 21 create for cloud mode # in 10249 ms
ok 22 multiple connection options are prevented # in 873 ms
ok 23 create collection # in 8542 ms
ok 24 create collection using solrUrl # in 5473 ms
ok 25 create collection using legacy solrUrl # in 5502 ms
ok 26 create collection using Zookeeper # in 5454 ms
ok 27 reject d option with invalid config dir # in 2753 ms
ok 28 accept d option with builtin config # in 5463 ms
ok 29 accept d option with explicit path to config # in 5397 ms
ok 30 accept n option as config name # in 5393 ms
ok 31 allow config reuse when n option specifies same config # in 9279 ms
ok 32 create multisharded collections when s provided # in 5309 ms
ok 33 create replicated collections when rf provided # in 6207 ms
ok 34 can delete collections # in 11749 ms
ok 35 can delete collections with solr-url # in 9767 ms
ok 36 collection delete also deletes zk config # in 7689 ms
ok 37 deletes accompanying zk config with nondefault name # in 7501 ms
ok 38 delete-config option can opt to leave config in zk # in 7459 ms
ok 39 start -e cloud works with --jvm-opts # in 23021 ms
ok 40 start -e cloud works with --no-prompt # in 21356 ms
ok 41 Check export command # in 20594 ms
ok 42 export fails on non cloud mode # in 10334 ms
Tika Server started on port 45210
ok 43 using curl to extract a single pdf file # in 21478 ms
ok 44 using the bin/solr post tool to extract content from pdf # in 21379 ms
ok 45 using the bin/solr post tool to crawl web site # in 23291 ms
Stopping Tika Server container
ok 46 healthcheck on cloud solr # in 11160 ms
ok 47 healthcheck errors on standalone solr # in 12201 ms
ok 48 solr help flag prints help # in 784 ms
ok 49 solr with no flags prints help # in 774 ms
ok 50 start help flag prints help # in 170 ms
ok 51 start h flag prints help # in 160 ms
ok 52 stop help flag prints help # in 171 ms
ok 53 restart help flag prints help # in 149 ms
ok 54 status help flag prints help # in 824 ms
ok 55 healthcheck help flag prints help # in 788 ms
ok 56 create help flag prints help # in 795 ms
ok 57 delete help flag prints help # in 766 ms
ok 58 zk help flag prints help # in 821 ms
ok 59 auth help flag prints help # in 785 ms
ok 60 assert help flag prints help # in 835 ms
ok 61 post help flag prints help # in 810 ms
ok 62 SQL Module # in 16575 ms
ok 63 icu collation in analysis-extras module # in 13455 ms
ok 64 Check lifecycle of sentiment classification # in 24476 ms
ok 65 lifecycle of package # in 8692 ms
ok 66 deploying and undeploying a collection level package # in 15874 ms
ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 68 Affinity placement plugin using sysprop # in 17200 ms
ok 69 Random placement plugin using ENV # in 16718 ms
ok 70 Check help command # in 2496 ms
ok 71 basic post with a type specified # in 6057 ms
ok 72 basic post with collection instead of url parameter # in 4636 ms
ok 73 basic post with solr-url and collection # in 4278 ms
ok 74 basic post WITHOUT a type specified # in 12654 ms
ok 75 crawling a directory as a dry-run # in 830 ms
ok 76 crawling a directory # in 4807 ms
ok 77 crawling a web site # in 34838 ms
ok 78 skipcommit and optimize and delete # in 4842 ms
ok 79 args mode # in 7759 ms
ok 80 stdin mode # in 4212 ms
not ok 81 verbose echo the Solr response # in 6797 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (13 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45205.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45205.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:45205/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c monitors_verbose -s http://localhost:45205 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-29 13:02:21.993; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-29 13:02:21.997; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:45205/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45205.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45205.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:45205/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c monitors_verbose -s http://localhost:45205 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-29 13:02:21.993; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-29 13:02:21.997; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:45205/solr/admin/collections
ok 82 post solr log into solr via cli # in 10587 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-04-29T13:02:54.432963159Z *** [WARN] *** Your open file limit is currently 1024.
2026-04-29T13:02:54.433007064Z It should be set to 65000 to avoid operational disruption.
2026-04-29T13:02:54.433010359Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-04-29T13:02:54.433349141Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-04-29T13:02:54.433366221Z
2026-04-29T13:02:54.443922328Z Java 17 detected. Enabled workaround for SOLR-16463
2026-04-29T13:02:54.451790967Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-04-29T13:02:54.502925159Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-04-29T13:02:54.541501231Z WARNING: A command line option has enabled the Security Manager
2026-04-29T13:02:54.541526152Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-04-29T13:02:55.922603464Z 2026-04-29 13:02:55.919 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-04-29T13:02:56.198519432Z 2026-04-29 13:02:56.197 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-04-29T13:02:56.224586972Z 2026-04-29 13:02:56.224 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-04-29T13:02:56.232176505Z 2026-04-29 13:02:56.231 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-04-29T13:02:56.232221384Z 2026-04-29 13:02:56.231 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-04-29T13:02:56.232359328Z 2026-04-29 13:02:56.231 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-04-29T13:02:56.232558432Z 2026-04-29 13:02:56.231 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-29T13:02:56.231967874Z
2026-04-29T13:02:56.235543579Z 2026-04-29 13:02:56.235 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-04-29T13:02:56.258093014Z 2026-04-29 13:02:56.257 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-04-29T13:02:56.272382482Z 2026-04-29 13:02:56.271 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-04-29T13:02:56.272424424Z 2026-04-29 13:02:56.271 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-04-29T13:02:56.311019050Z 2026-04-29 13:02:56.310 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-04-29T13:02:57.062789315Z 2026-04-29 13:02:57.062 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-04-29T13:02:57.311221731Z 2026-04-29 13:02:57.310 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-04-29T13:02:57.315953765Z 2026-04-29 13:02:57.315 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-04-29T13:02:57.316026486Z 2026-04-29 13:02:57.315 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-04-29T13:02:57.318469243Z 2026-04-29 13:02:57.318 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-04-29T13:02:57.318545804Z 2026-04-29 13:02:57.318 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-04-29T13:02:57.354411496Z 2026-04-29 13:02:57.353 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-04-29T13:02:57.824780110Z 2026-04-29 13:02:57.822 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-04-29T13:02:57.835662240Z 2026-04-29 13:02:57.835 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-29T13:02:57.889016746Z 2026-04-29 13:02:57.888 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-04-29T13:02:58.069591794Z 2026-04-29 13:02:58.067 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-29T13:02:58.069618764Z 2026-04-29 13:02:58.067 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-29T13:02:58.255952288Z 2026-04-29 13:02:58.255 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-04-29T13:02:58.283706783Z 2026-04-29 13:02:58.283 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-29T13:02:58.307912223Z 2026-04-29 13:02:58.307 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-04-29T13:02:58.321789318Z 2026-04-29 13:02:58.321 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72144021442527232-solr-node1:8983_solr-n_0000000000) starting
2026-04-29T13:02:58.400299739Z 2026-04-29 13:02:58.399 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-04-29T13:02:58.401846921Z 2026-04-29 13:02:58.401 INFO (OverseerStateUpdate-72144021442527232-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-04-29T13:02:58.420806161Z 2026-04-29 13:02:58.417 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-04-29T13:02:58.462366680Z 2026-04-29 13:02:58.461 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-04-29T13:02:58.926349841Z 2026-04-29 13:02:58.925 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-04-29T13:02:59.127110511Z 2026-04-29 13:02:59.126 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-04-29T13:02:59.479761556Z 2026-04-29 13:02:59.479 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-29T13:02:59.629878973Z 2026-04-29 13:02:59.628 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-04-29T13:02:59.635509948Z 2026-04-29 13:02:59.635 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_04_29.request.log
2026-04-29T13:02:59.641040191Z 2026-04-29 13:02:59.640 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-04-29T13:02:59.778863751Z 2026-04-29 13:02:59.642 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5191ms
2026-04-29T13:02:59.975409450Z 2026-04-29 13:02:59.974 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=55
2026-04-29T13:03:00.042267044Z 2026-04-29 13:03:00.041 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13
2026-04-29T13:03:03.837193919Z 2026-04-29 13:03:03.836 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-04-29T13:03:04.386376998Z 2026-04-29 13:03:04.385 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-04-29T13:03:08.028040526Z 2026-04-29 13:03:08.027 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-04-29T13:03:08.059474377Z 2026-04-29 13:03:08.058 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=9
2026-04-29T13:03:08.080848145Z 2026-04-29 13:03:08.080 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-04-29T13:03:08.575154767Z 2026-04-29 13:03:08.573 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-04-29T13:03:08.594165184Z 2026-04-29 13:03:08.593 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-04-29T13:03:08.599364357Z 2026-04-29 13:03:08.598 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:08.642910408Z 2026-04-29 13:03:08.642 INFO (OverseerStateUpdate-72144021442527232-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-04-29T13:03:08.642936578Z "core":"test-collection_shard2_replica_n1",
2026-04-29T13:03:08.642941002Z "node_name":"solr-node3:8985_solr",
2026-04-29T13:03:08.642945293Z "base_url":"http://solr-node3:8985/solr",
2026-04-29T13:03:08.642948627Z "collection":"test-collection",
2026-04-29T13:03:08.642951953Z "shard":"shard2",
2026-04-29T13:03:08.642955339Z "state":"down",
2026-04-29T13:03:08.642959135Z "type":"NRT",
2026-04-29T13:03:08.642962464Z "operation":"ADDREPLICA",
2026-04-29T13:03:08.642965812Z "waitForFinalState":"false"}
2026-04-29T13:03:08.650956964Z 2026-04-29 13:03:08.650 INFO (OverseerStateUpdate-72144021442527232-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-04-29T13:03:08.650988979Z "core":"test-collection_shard3_replica_n2",
2026-04-29T13:03:08.650992305Z "node_name":"solr-node1:8983_solr",
2026-04-29T13:03:08.650994994Z "base_url":"http://solr-node1:8983/solr",
2026-04-29T13:03:08.650997556Z "collection":"test-collection",
2026-04-29T13:03:08.651000151Z "shard":"shard3",
2026-04-29T13:03:08.651002659Z "state":"down",
2026-04-29T13:03:08.651005196Z "type":"NRT",
2026-04-29T13:03:08.651007722Z "operation":"ADDREPLICA",
2026-04-29T13:03:08.651010372Z "waitForFinalState":"false"}
2026-04-29T13:03:08.658899125Z 2026-04-29 13:03:08.658 INFO (OverseerStateUpdate-72144021442527232-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-04-29T13:03:08.658917957Z "core":"test-collection_shard1_replica_n4",
2026-04-29T13:03:08.658921501Z "node_name":"solr-node2:8984_solr",
2026-04-29T13:03:08.658937112Z "base_url":"http://solr-node2:8984/solr",
2026-04-29T13:03:08.658939298Z "collection":"test-collection",
2026-04-29T13:03:08.658941266Z "shard":"shard1",
2026-04-29T13:03:08.658943182Z "state":"down",
2026-04-29T13:03:08.658945048Z "type":"NRT",
2026-04-29T13:03:08.658947622Z "operation":"ADDREPLICA",
2026-04-29T13:03:08.658950587Z "waitForFinalState":"false"}
2026-04-29T13:03:08.767887361Z 2026-04-29 13:03:08.767 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:08.830613267Z 2026-04-29 13:03:08.830 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard3_replica_n2&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-04-29T13:03:08.870597401Z 2026-04-29 13:03:08.870 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-04-29T13:03:08.926395300Z 2026-04-29 13:03:08.925 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-04-29T13:03:09.278159080Z 2026-04-29 13:03:09.275 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-04-29T13:03:09.278225503Z 2026-04-29 13:03:09.275 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n2' using configuration from configset techproducts, trusted=true
2026-04-29T13:03:09.318254457Z 2026-04-29 13:03:09.317 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n2], dataDir=[/var/solr/data/test-collection_shard3_replica_n2/data/]
2026-04-29T13:03:10.134019759Z 2026-04-29 13:03:10.132 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-29T13:03:10.192987565Z 2026-04-29 13:03:10.191 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-04-29T13:03:10.201976741Z 2026-04-29 13:03:10.201 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-04-29T13:03:10.203207173Z 2026-04-29 13:03:10.202 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-04-29T13:03:10.203608237Z 2026-04-29 13:03:10.202 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-04-29T13:03:10.300673755Z 2026-04-29 13:03:10.299 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-04-29T13:03:10.302859346Z 2026-04-29 13:03:10.302 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-04-29T13:03:10.304894568Z 2026-04-29 13:03:10.304 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-04-29T13:03:10.307235392Z 2026-04-29 13:03:10.306 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-04-29T13:03:10.324196879Z 2026-04-29 13:03:10.323 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-04-29T13:03:10.333096877Z 2026-04-29 13:03:10.332 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-04-29T13:03:10.342772846Z 2026-04-29 13:03:10.342 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n2 solr-node1-6 core_node5 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-04-29T13:03:10.342794421Z 2026-04-29 13:03:10.342 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n2 solr-node1-6 core_node5 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-04-29T13:03:10.352659052Z 2026-04-29 13:03:10.352 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n2 solr-node1-6 core_node5 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-04-29T13:03:10.369519803Z 2026-04-29 13:03:10.368 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-04-29T13:03:10.371505848Z 2026-04-29 13:03:10.371 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-04-29T13:03:10.385646783Z 2026-04-29 13:03:10.385 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-04-29T13:03:10.385667111Z 2026-04-29 13:03:10.385 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-04-29T13:03:10.385670160Z 2026-04-29 13:03:10.385 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n2/
2026-04-29T13:03:10.387038299Z 2026-04-29 13:03:10.386 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-04-29T13:03:10.387550329Z 2026-04-29 13:03:10.387 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n2/ has no replicas
2026-04-29T13:03:10.389899157Z 2026-04-29 13:03:10.389 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72144021442527232-core_node5-n_0000000000
2026-04-29T13:03:10.400207024Z 2026-04-29 13:03:10.399 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n2/ shard3
2026-04-29T13:03:10.508650652Z 2026-04-29 13:03:10.507 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:10.513590117Z 2026-04-29 13:03:10.513 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node5 x:test-collection_shard3_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-04-29T13:03:10.523541934Z 2026-04-29 13:03:10.522 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1693
2026-04-29T13:03:10.621939269Z 2026-04-29 13:03:10.621 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2265355,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-04-29T13:02:54.18839583Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-04-29T13:03:00.389920363Z *** [WARN] *** Your open file limit is currently 1024.
2026-04-29T13:03:00.389958704Z It should be set to 65000 to avoid operational disruption.
2026-04-29T13:03:00.389963006Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-04-29T13:03:00.390280761Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-04-29T13:03:00.390286339Z
2026-04-29T13:03:00.400385171Z Java 17 detected. Enabled workaround for SOLR-16463
2026-04-29T13:03:00.408625212Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-04-29T13:03:00.454604126Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-04-29T13:03:00.488224303Z WARNING: A command line option has enabled the Security Manager
2026-04-29T13:03:00.488251092Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-04-29T13:03:01.822558033Z 2026-04-29 13:03:01.816 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-04-29T13:03:02.208634546Z 2026-04-29 13:03:02.207 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-04-29T13:03:02.240361093Z 2026-04-29 13:03:02.239 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-04-29T13:03:02.248366488Z 2026-04-29 13:03:02.247 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-04-29T13:03:02.248440191Z 2026-04-29 13:03:02.247 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-04-29T13:03:02.248643485Z 2026-04-29 13:03:02.247 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-04-29T13:03:02.248940516Z 2026-04-29 13:03:02.248 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-29T13:03:02.247991598Z
2026-04-29T13:03:02.252483274Z 2026-04-29 13:03:02.251 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-04-29T13:03:02.279493208Z 2026-04-29 13:03:02.278 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-04-29T13:03:02.307169525Z 2026-04-29 13:03:02.306 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-29T13:03:02.474214270Z 2026-04-29 13:03:02.473 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-04-29T13:03:02.499625709Z 2026-04-29 13:03:02.499 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-29T13:03:02.499753771Z 2026-04-29 13:03:02.499 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-29T13:03:02.499895802Z 2026-04-29 13:03:02.499 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-29T13:03:02.612431601Z 2026-04-29 13:03:02.611 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-04-29T13:03:02.612514867Z 2026-04-29 13:03:02.612 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-04-29T13:03:02.612752784Z 2026-04-29 13:03:02.612 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-04-29T13:03:02.651485326Z 2026-04-29 13:03:02.650 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-04-29T13:03:03.531502966Z 2026-04-29 13:03:03.530 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-04-29T13:03:03.679504331Z 2026-04-29 13:03:03.678 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-04-29T13:03:03.681041810Z 2026-04-29 13:03:03.679 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-04-29T13:03:03.715641969Z 2026-04-29 13:03:03.714 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-29T13:03:03.724890506Z 2026-04-29 13:03:03.724 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-04-29T13:03:03.731008082Z 2026-04-29 13:03:03.730 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-29T13:03:03.731168676Z 2026-04-29 13:03:03.730 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-29T13:03:03.768546086Z 2026-04-29 13:03:03.767 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-04-29T13:03:03.778607121Z 2026-04-29 13:03:03.777 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-04-29T13:03:03.799921719Z 2026-04-29 13:03:03.799 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-29T13:03:03.812104779Z 2026-04-29 13:03:03.811 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-04-29T13:03:03.824280698Z 2026-04-29 13:03:03.823 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-04-29T13:03:03.837117073Z 2026-04-29 13:03:03.836 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-04-29T13:03:03.927491048Z 2026-04-29 13:03:03.926 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-04-29T13:03:04.264051703Z 2026-04-29 13:03:04.263 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-04-29T13:03:04.386919148Z 2026-04-29 13:03:04.386 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-04-29T13:03:04.465936200Z 2026-04-29 13:03:04.465 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-04-29T13:03:04.768939705Z 2026-04-29 13:03:04.768 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-29T13:03:04.924511736Z 2026-04-29 13:03:04.923 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-04-29T13:03:04.930114990Z 2026-04-29 13:03:04.929 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_04_29.request.log
2026-04-29T13:03:04.934871034Z 2026-04-29 13:03:04.934 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-04-29T13:03:04.935762201Z 2026-04-29 13:03:04.935 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4528ms
2026-04-29T13:03:05.185056922Z 2026-04-29 13:03:05.184 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=64
2026-04-29T13:03:05.258573441Z 2026-04-29 13:03:05.257 INFO (qtp985324122-56-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-04-29T13:03:08.831821159Z 2026-04-29 13:03:08.831 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-04-29T13:03:08.883151800Z 2026-04-29 13:03:08.882 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-04-29T13:03:08.958935721Z 2026-04-29 13:03:08.958 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-04-29T13:03:09.305503732Z 2026-04-29 13:03:09.304 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-04-29T13:03:09.305527374Z 2026-04-29 13:03:09.305 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-04-29T13:03:09.326225288Z 2026-04-29 13:03:09.325 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-04-29T13:03:10.021832118Z 2026-04-29 13:03:10.021 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-29T13:03:10.042902549Z 2026-04-29 13:03:10.042 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-04-29T13:03:10.051641076Z 2026-04-29 13:03:10.051 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-04-29T13:03:10.053186666Z 2026-04-29 13:03:10.052 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-04-29T13:03:10.053256124Z 2026-04-29 13:03:10.052 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-04-29T13:03:10.112370065Z 2026-04-29 13:03:10.111 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-04-29T13:03:10.117065464Z 2026-04-29 13:03:10.116 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-04-29T13:03:10.119445547Z 2026-04-29 13:03:10.119 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-04-29T13:03:10.121670150Z 2026-04-29 13:03:10.121 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-04-29T13:03:10.137398062Z 2026-04-29 13:03:10.136 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-04-29T13:03:10.146672507Z 2026-04-29 13:03:10.146 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-04-29T13:03:10.154012974Z 2026-04-29 13:03:10.153 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-04-29T13:03:10.154029925Z 2026-04-29 13:03:10.153 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-04-29T13:03:10.162564329Z 2026-04-29 13:03:10.162 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-04-29T13:03:10.185656228Z 2026-04-29 13:03:10.184 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-04-29T13:03:10.187251067Z 2026-04-29 13:03:10.186 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-04-29T13:03:10.207113576Z 2026-04-29 13:03:10.206 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-04-29T13:03:10.207134914Z 2026-04-29 13:03:10.206 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-04-29T13:03:10.207143613Z 2026-04-29 13:03:10.206 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-04-29T13:03:10.208839524Z 2026-04-29 13:03:10.208 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-04-29T13:03:10.209440535Z 2026-04-29 13:03:10.209 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-04-29T13:03:10.211180343Z 2026-04-29 13:03:10.210 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72144021442527235-core_node6-n_0000000000
2026-04-29T13:03:10.225749001Z 2026-04-29 13:03:10.225 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-04-29T13:03:10.342043836Z 2026-04-29 13:03:10.341 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:10.347712207Z 2026-04-29 13:03:10.347 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-04-29T13:03:10.361149246Z 2026-04-29 13:03:10.360 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1530
2026-04-29T13:03:10.508621502Z 2026-04-29 13:03:10.508 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:10.621938009Z 2026-04-29 13:03:10.621 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2265761,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-04-29T13:03:00.154787086Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-04-29T13:03:00.642076713Z *** [WARN] *** Your open file limit is currently 1024.
2026-04-29T13:03:00.642115807Z It should be set to 65000 to avoid operational disruption.
2026-04-29T13:03:00.642119339Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-04-29T13:03:00.642652733Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-04-29T13:03:00.642658387Z
2026-04-29T13:03:00.655399018Z Java 17 detected. Enabled workaround for SOLR-16463
2026-04-29T13:03:00.664201265Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-04-29T13:03:00.711377365Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-04-29T13:03:00.745688269Z WARNING: A command line option has enabled the Security Manager
2026-04-29T13:03:00.745716775Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-04-29T13:03:02.191032081Z 2026-04-29 13:03:02.187 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-04-29T13:03:02.491978014Z 2026-04-29 13:03:02.491 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-04-29T13:03:02.522959791Z 2026-04-29 13:03:02.522 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-04-29T13:03:02.530259687Z 2026-04-29 13:03:02.529 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-04-29T13:03:02.530457122Z 2026-04-29 13:03:02.529 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-04-29T13:03:02.530626648Z 2026-04-29 13:03:02.529 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-04-29T13:03:02.530854609Z 2026-04-29 13:03:02.529 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-29T13:03:02.529881051Z
2026-04-29T13:03:02.533306245Z 2026-04-29 13:03:02.532 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-04-29T13:03:02.557720354Z 2026-04-29 13:03:02.557 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-04-29T13:03:02.583185360Z 2026-04-29 13:03:02.582 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-29T13:03:02.747274942Z 2026-04-29 13:03:02.746 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-04-29T13:03:02.770379817Z 2026-04-29 13:03:02.769 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-29T13:03:02.770559012Z 2026-04-29 13:03:02.769 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-29T13:03:02.770675569Z 2026-04-29 13:03:02.770 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-29T13:03:02.881842622Z 2026-04-29 13:03:02.881 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-04-29T13:03:02.882007201Z 2026-04-29 13:03:02.881 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-04-29T13:03:02.882103980Z 2026-04-29 13:03:02.881 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-04-29T13:03:02.920349672Z 2026-04-29 13:03:02.919 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-04-29T13:03:04.105394265Z 2026-04-29 13:03:04.104 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-04-29T13:03:04.252264707Z 2026-04-29 13:03:04.251 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-04-29T13:03:04.253849550Z 2026-04-29 13:03:04.251 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-04-29T13:03:04.263042991Z 2026-04-29 13:03:04.262 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-29T13:03:04.268283102Z 2026-04-29 13:03:04.267 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-04-29T13:03:04.271970255Z 2026-04-29 13:03:04.271 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-29T13:03:04.272136500Z 2026-04-29 13:03:04.271 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-29T13:03:04.318028353Z 2026-04-29 13:03:04.317 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-04-29T13:03:04.326948549Z 2026-04-29 13:03:04.326 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-04-29T13:03:04.350183090Z 2026-04-29 13:03:04.349 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-29T13:03:04.363525348Z 2026-04-29 13:03:04.362 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-04-29T13:03:04.375173210Z 2026-04-29 13:03:04.374 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-04-29T13:03:04.387536642Z 2026-04-29 13:03:04.387 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-04-29T13:03:04.435495687Z 2026-04-29 13:03:04.434 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-04-29T13:03:04.757445420Z 2026-04-29 13:03:04.756 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-04-29T13:03:04.952340807Z 2026-04-29 13:03:04.951 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-04-29T13:03:05.221922601Z 2026-04-29 13:03:05.221 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-29T13:03:05.350233308Z 2026-04-29 13:03:05.349 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-04-29T13:03:05.355876874Z 2026-04-29 13:03:05.355 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_04_29.request.log
2026-04-29T13:03:05.359387794Z 2026-04-29 13:03:05.358 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-04-29T13:03:05.360331810Z 2026-04-29 13:03:05.360 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4697ms
2026-04-29T13:03:06.755291766Z 2026-04-29 13:03:06.749 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=65
2026-04-29T13:03:06.841127974Z 2026-04-29 13:03:06.840 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=27
2026-04-29T13:03:08.539271227Z 2026-04-29 13:03:08.538 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=9
2026-04-29T13:03:08.834451919Z 2026-04-29 13:03:08.833 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-04-29T13:03:08.902961624Z 2026-04-29 13:03:08.901 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-04-29T13:03:09.017622725Z 2026-04-29 13:03:09.016 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-04-29T13:03:09.381863946Z 2026-04-29 13:03:09.381 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-04-29T13:03:09.381890866Z 2026-04-29 13:03:09.381 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n1' using configuration from configset techproducts, trusted=true
2026-04-29T13:03:09.401546777Z 2026-04-29 13:03:09.401 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n1], dataDir=[/var/solr/data/test-collection_shard2_replica_n1/data/]
2026-04-29T13:03:09.915727327Z 2026-04-29 13:03:09.914 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-29T13:03:09.945186049Z 2026-04-29 13:03:09.944 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-04-29T13:03:09.954370054Z 2026-04-29 13:03:09.953 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-04-29T13:03:09.955522520Z 2026-04-29 13:03:09.954 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-04-29T13:03:09.955572415Z 2026-04-29 13:03:09.955 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-04-29T13:03:10.006682799Z 2026-04-29 13:03:10.006 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-04-29T13:03:10.014328818Z 2026-04-29 13:03:10.013 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-04-29T13:03:10.016755150Z 2026-04-29 13:03:10.016 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-04-29T13:03:10.021093628Z 2026-04-29 13:03:10.020 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-04-29T13:03:10.033277091Z 2026-04-29 13:03:10.032 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-04-29T13:03:10.042404309Z 2026-04-29 13:03:10.041 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-04-29T13:03:10.049424382Z 2026-04-29 13:03:10.048 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-04-29T13:03:10.050090484Z 2026-04-29 13:03:10.049 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-04-29T13:03:10.057373094Z 2026-04-29 13:03:10.056 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-04-29T13:03:10.075876584Z 2026-04-29 13:03:10.074 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-04-29T13:03:10.077604857Z 2026-04-29 13:03:10.077 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-04-29T13:03:10.097758809Z 2026-04-29 13:03:10.097 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-04-29T13:03:10.097846029Z 2026-04-29 13:03:10.097 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-04-29T13:03:10.097906649Z 2026-04-29 13:03:10.097 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n1/
2026-04-29T13:03:10.100359195Z 2026-04-29 13:03:10.099 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-04-29T13:03:10.100431195Z 2026-04-29 13:03:10.100 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ has no replicas
2026-04-29T13:03:10.101891034Z 2026-04-29 13:03:10.101 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72144021442527236-core_node3-n_0000000000
2026-04-29T13:03:10.116994879Z 2026-04-29 13:03:10.116 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ shard2
2026-04-29T13:03:10.224423409Z 2026-04-29 13:03:10.223 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:10.229843994Z 2026-04-29 13:03:10.229 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-04-29T13:03:10.240119290Z 2026-04-29 13:03:10.239 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1407
2026-04-29T13:03:10.342951436Z 2026-04-29 13:03:10.341 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:10.508570370Z 2026-04-29 13:03:10.508 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:10.542322407Z 2026-04-29 13:03:10.541 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-04-29T13:03:10.621968978Z 2026-04-29 13:03:10.621 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-29T13:03:10.626648745Z 2026-04-29 13:03:10.626 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2071
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2265938,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-04-29T13:03:00.410985789Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 29649 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 66b76b382631: Waiting
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# c63139dd1378: Pulling fs layer
# bcc055df9b8e: Pulling fs layer
# f8594c2ad62e: Pulling fs layer
# 8c1ec5f7d4c5: Pulling fs layer
# ef7b8d9674bf: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 8c1ec5f7d4c5: Waiting
# ef7b8d9674bf: Waiting
# 4f4fb700ef54: Waiting
# f8594c2ad62e: Verifying Checksum
# f8594c2ad62e: Download complete
# bcc055df9b8e: Verifying Checksum
# bcc055df9b8e: Download complete
# ef7b8d9674bf: Verifying Checksum
# ef7b8d9674bf: Download complete
# 8c1ec5f7d4c5: Verifying Checksum
# 8c1ec5f7d4c5: Download complete
# 4f4fb700ef54: Verifying Checksum
# c63139dd1378: Verifying Checksum
# c63139dd1378: Download complete
# c63139dd1378: Pull complete
# bcc055df9b8e: Pull complete
# f8594c2ad62e: Pull complete
# 8c1ec5f7d4c5: Pull complete
# ef7b8d9674bf: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:3c899e5dcf8ad3da6f6c5fe90503c273cf15b5825eecb38d204d657279f9552f
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 3fea0d942d812e7cc13ee141395fecf63234905b1f59c8e911a78a0a41da3d72
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 7011fb11e48020c2207d94a07503c9b46cf3a825c0bdad56927df3f3eff7d82d
# Starting solr-node2 connected to first node's ZooKeeper
# 8d8a9d9520f1c4a8884c0c8641a2275ac4a48a2347e01de1e375dae6fe533259
# Started solr-node3 connected to first node's ZooKeeper
# a4b52e12f4a7663c9d945222598d0f23bdffe69b3030ff3846f9ec52a9eadd1e
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-04-29 13:03:08.121; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-29 13:03:08.159; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 16679 ms
ok 85 snapshot lifecycle # in 5217 ms
ok 86 snapshot list # in 3603 ms
ok 87 snapshot describe # in 3608 ms
ok 88 start solr with ssl # in 18974 ms
ok 89 use different hostname when not checking peer-name # in 26821 ms
ok 90 start solr with ssl and auth # in 15973 ms
ok 91 start solr with client truststore and security manager # in 18858 ms
ok 92 start solr with mTLS needed # in 48038 ms
ok 93 start solr with mTLS wanted # in 37550 ms
ok 94 test keystore reload # in 63144 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 16724 ms
ok 96 stop command for single port # in 20265 ms
ok 97 check stop command doesn't hang # in 244 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8578 ms
ok 99 deprecated system properties converted to modern properties # in 195555 ms
ok 100 start with custom jetty options # in 8770 ms
ok 101 bootstrapping a configset # in 10024 ms
ok 102 status detects locally running solr # in 10950 ms
ok 103 status with --solr-url from user # in 9125 ms
ok 104 status with --port from user # in 8688 ms
ok 105 multiple connection options are prevented # in 945 ms
ok 106 status with invalid --solr-url from user # in 1364 ms
ok 107 status with --short format # in 196240 ms
ok 108 searching solr via locally executed streaming expression # in 1888 ms
ok 109 searching solr via remotely executed streaming expression # in 1579 ms
ok 110 variable interpolation # in 1987 ms
ok 111 searching solr without credentials fails with error # in 1253 ms
ok 112 --version returns Solr version # in 850 ms
ok 113 version as direct tool call still runs # in 808 ms
ok 114 short help # in 999 ms
ok 115 short help is inferred # in 915 ms
ok 116 long help # in 882 ms
ok 117 running subcommands with zk is prevented # in 820 ms
ok 118 listing out files # in 2299 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 8335 ms
ok 120 copying files around # in 11520 ms
ok 121 upconfig # in 2655 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1247 ms
ok 123 downconfig # in 1216 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1518 ms
ok 125 env var ZK_HOST is honored # in 3320 ms
ok 126 Cleanup # in 1889 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 20s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/b22ewhvonlpm2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 193 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/193/
No tests ran.
Build Log:
[...truncated 420 lines...]
# ERROR: Server refused connection at: http://localhost:34323/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:34323/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Posting files to [base] url http://localhost:34323/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:34323/solr/monitors_verbose/update...
#
# ERROR: Server refused connection at: http://localhost:34323/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:34323/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 82 post solr log into solr via cli # in 8464 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-04-28T17:54:44.577676950Z *** [WARN] *** Your open file limit is currently 1024.
2026-04-28T17:54:44.577738790Z It should be set to 65000 to avoid operational disruption.
2026-04-28T17:54:44.577744095Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-04-28T17:54:44.578003334Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-04-28T17:54:44.578014255Z
2026-04-28T17:54:44.586722246Z Java 17 detected. Enabled workaround for SOLR-16463
2026-04-28T17:54:44.594017389Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-04-28T17:54:44.636477915Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-04-28T17:54:44.665959533Z WARNING: A command line option has enabled the Security Manager
2026-04-28T17:54:44.665976342Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-04-28T17:54:45.816765608Z 2026-04-28 17:54:45.813 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-04-28T17:54:46.114767442Z 2026-04-28 17:54:46.114 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-04-28T17:54:46.143262279Z 2026-04-28 17:54:46.142 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-04-28T17:54:46.151347534Z 2026-04-28 17:54:46.150 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-04-28T17:54:46.151481318Z 2026-04-28 17:54:46.150 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-04-28T17:54:46.151606238Z 2026-04-28 17:54:46.150 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-04-28T17:54:46.151769588Z 2026-04-28 17:54:46.151 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-28T17:54:46.151017654Z
2026-04-28T17:54:46.154575148Z 2026-04-28 17:54:46.154 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-04-28T17:54:46.179462559Z 2026-04-28 17:54:46.178 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-04-28T17:54:46.194771285Z 2026-04-28 17:54:46.194 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-04-28T17:54:46.194878711Z 2026-04-28 17:54:46.194 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-04-28T17:54:46.232955685Z 2026-04-28 17:54:46.232 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-04-28T17:54:47.227368547Z 2026-04-28 17:54:47.226 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-04-28T17:54:47.522982535Z 2026-04-28 17:54:47.521 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-04-28T17:54:47.530005398Z 2026-04-28 17:54:47.529 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-04-28T17:54:47.530131407Z 2026-04-28 17:54:47.529 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-04-28T17:54:47.534086785Z 2026-04-28 17:54:47.533 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-04-28T17:54:47.534184879Z 2026-04-28 17:54:47.533 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-04-28T17:54:47.583378869Z 2026-04-28 17:54:47.582 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-04-28T17:54:48.037233024Z 2026-04-28 17:54:48.036 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-04-28T17:54:48.054642921Z 2026-04-28 17:54:48.053 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-28T17:54:48.119351911Z 2026-04-28 17:54:48.118 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-04-28T17:54:48.151370813Z 2026-04-28 17:54:48.150 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-28T17:54:48.151459469Z 2026-04-28 17:54:48.150 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-28T17:54:48.301850621Z 2026-04-28 17:54:48.300 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-04-28T17:54:48.328747655Z 2026-04-28 17:54:48.328 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-28T17:54:48.345204442Z 2026-04-28 17:54:48.344 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-04-28T17:54:48.353273602Z 2026-04-28 17:54:48.352 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72139557998690304-solr-node1:8983_solr-n_0000000000) starting
2026-04-28T17:54:48.430339678Z 2026-04-28 17:54:48.429 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-04-28T17:54:48.430864719Z 2026-04-28 17:54:48.430 INFO (OverseerStateUpdate-72139557998690304-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-04-28T17:54:48.444987213Z 2026-04-28 17:54:48.442 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-04-28T17:54:48.491881049Z 2026-04-28 17:54:48.491 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-04-28T17:54:48.947929238Z 2026-04-28 17:54:48.947 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-04-28T17:54:49.154448055Z 2026-04-28 17:54:49.153 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-04-28T17:54:49.450426668Z 2026-04-28 17:54:49.449 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-28T17:54:49.600457376Z 2026-04-28 17:54:49.599 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-04-28T17:54:49.607225136Z 2026-04-28 17:54:49.606 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_04_28.request.log
2026-04-28T17:54:49.612367395Z 2026-04-28 17:54:49.611 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-04-28T17:54:49.647926019Z 2026-04-28 17:54:49.646 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5053ms
2026-04-28T17:54:49.849854598Z 2026-04-28 17:54:49.849 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=62
2026-04-28T17:54:49.909366517Z 2026-04-28 17:54:49.908 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-04-28T17:54:52.997587953Z 2026-04-28 17:54:52.997 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-04-28T17:54:57.203855818Z 2026-04-28 17:54:57.203 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-04-28T17:54:59.248960303Z 2026-04-28 17:54:59.248 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-04-28T17:54:59.271265521Z 2026-04-28 17:54:59.270 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=7
2026-04-28T17:54:59.293583514Z 2026-04-28 17:54:59.293 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1
2026-04-28T17:54:59.534887827Z 2026-04-28 17:54:59.532 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-04-28T17:54:59.549902422Z 2026-04-28 17:54:59.549 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-04-28T17:54:59.555789800Z 2026-04-28 17:54:59.555 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:54:59.594370311Z 2026-04-28 17:54:59.593 INFO (OverseerStateUpdate-72139557998690304-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-04-28T17:54:59.594388825Z "core":"test-collection_shard2_replica_n1",
2026-04-28T17:54:59.594391945Z "node_name":"solr-node3:8985_solr",
2026-04-28T17:54:59.594394706Z "base_url":"http://solr-node3:8985/solr",
2026-04-28T17:54:59.594397398Z "collection":"test-collection",
2026-04-28T17:54:59.594400090Z "shard":"shard2",
2026-04-28T17:54:59.594402566Z "state":"down",
2026-04-28T17:54:59.594404938Z "type":"NRT",
2026-04-28T17:54:59.594407416Z "operation":"ADDREPLICA",
2026-04-28T17:54:59.594409861Z "waitForFinalState":"false"}
2026-04-28T17:54:59.600119004Z 2026-04-28 17:54:59.599 INFO (OverseerStateUpdate-72139557998690304-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-04-28T17:54:59.600132530Z "core":"test-collection_shard1_replica_n2",
2026-04-28T17:54:59.600135732Z "node_name":"solr-node2:8984_solr",
2026-04-28T17:54:59.600138358Z "base_url":"http://solr-node2:8984/solr",
2026-04-28T17:54:59.600140845Z "collection":"test-collection",
2026-04-28T17:54:59.600143360Z "shard":"shard1",
2026-04-28T17:54:59.600145919Z "state":"down",
2026-04-28T17:54:59.600148321Z "type":"NRT",
2026-04-28T17:54:59.600150761Z "operation":"ADDREPLICA",
2026-04-28T17:54:59.600153528Z "waitForFinalState":"false"}
2026-04-28T17:54:59.604750536Z 2026-04-28 17:54:59.604 INFO (OverseerStateUpdate-72139557998690304-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-04-28T17:54:59.604766792Z "core":"test-collection_shard3_replica_n4",
2026-04-28T17:54:59.604770118Z "node_name":"solr-node1:8983_solr",
2026-04-28T17:54:59.604783656Z "base_url":"http://solr-node1:8983/solr",
2026-04-28T17:54:59.604785958Z "collection":"test-collection",
2026-04-28T17:54:59.604788229Z "shard":"shard3",
2026-04-28T17:54:59.604790200Z "state":"down",
2026-04-28T17:54:59.604792118Z "type":"NRT",
2026-04-28T17:54:59.604794112Z "operation":"ADDREPLICA",
2026-04-28T17:54:59.604796048Z "waitForFinalState":"false"}
2026-04-28T17:54:59.715459020Z 2026-04-28 17:54:59.715 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:54:59.792114986Z 2026-04-28 17:54:59.791 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-04-28T17:54:59.869676529Z 2026-04-28 17:54:59.869 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-04-28T17:54:59.915324317Z 2026-04-28 17:54:59.914 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-04-28T17:55:00.226487044Z 2026-04-28 17:55:00.225 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-04-28T17:55:00.226509639Z 2026-04-28 17:55:00.226 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n4' using configuration from configset techproducts, trusted=true
2026-04-28T17:55:00.253307368Z 2026-04-28 17:55:00.252 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n4], dataDir=[/var/solr/data/test-collection_shard3_replica_n4/data/]
2026-04-28T17:55:01.014605324Z 2026-04-28 17:55:01.013 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-28T17:55:01.069042104Z 2026-04-28 17:55:01.068 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-04-28T17:55:01.074360423Z 2026-04-28 17:55:01.074 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-04-28T17:55:01.075644599Z 2026-04-28 17:55:01.074 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-04-28T17:55:01.075775832Z 2026-04-28 17:55:01.075 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-04-28T17:55:01.214291602Z 2026-04-28 17:55:01.213 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-04-28T17:55:01.219257360Z 2026-04-28 17:55:01.218 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-04-28T17:55:01.220981534Z 2026-04-28 17:55:01.220 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-04-28T17:55:01.222558906Z 2026-04-28 17:55:01.222 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-04-28T17:55:01.243455193Z 2026-04-28 17:55:01.242 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-04-28T17:55:01.258548013Z 2026-04-28 17:55:01.257 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-04-28T17:55:01.262973464Z 2026-04-28 17:55:01.262 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-04-28T17:55:01.263023483Z 2026-04-28 17:55:01.262 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-04-28T17:55:01.273096410Z 2026-04-28 17:55:01.272 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-04-28T17:55:01.277456851Z 2026-04-28 17:55:01.276 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-04-28T17:55:01.278722481Z 2026-04-28 17:55:01.278 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-04-28T17:55:01.292890619Z 2026-04-28 17:55:01.289 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-04-28T17:55:01.292984409Z 2026-04-28 17:55:01.289 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-04-28T17:55:01.293058231Z 2026-04-28 17:55:01.289 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n4/
2026-04-28T17:55:01.293074831Z 2026-04-28 17:55:01.290 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-04-28T17:55:01.293145148Z 2026-04-28 17:55:01.290 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ has no replicas
2026-04-28T17:55:01.293210841Z 2026-04-28 17:55:01.292 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72139557998690304-core_node6-n_0000000000
2026-04-28T17:55:01.297136921Z 2026-04-28 17:55:01.296 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ shard3
2026-04-28T17:55:01.403848748Z 2026-04-28 17:55:01.403 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:01.406622521Z 2026-04-28 17:55:01.406 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-04-28T17:55:01.413970204Z 2026-04-28 17:55:01.413 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1622
2026-04-28T17:55:01.514044859Z 2026-04-28 17:55:01.513 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2008573,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-04-28T17:54:44.372554388Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-04-28T17:54:50.188492267Z *** [WARN] *** Your open file limit is currently 1024.
2026-04-28T17:54:50.188513303Z It should be set to 65000 to avoid operational disruption.
2026-04-28T17:54:50.188516200Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-04-28T17:54:50.188835795Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-04-28T17:54:50.188848459Z
2026-04-28T17:54:50.197539973Z Java 17 detected. Enabled workaround for SOLR-16463
2026-04-28T17:54:50.204992836Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-04-28T17:54:50.246605354Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-04-28T17:54:50.277160341Z WARNING: A command line option has enabled the Security Manager
2026-04-28T17:54:50.277177191Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-04-28T17:54:51.383060269Z 2026-04-28 17:54:51.380 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-04-28T17:54:51.632738225Z 2026-04-28 17:54:51.632 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-04-28T17:54:51.655662007Z 2026-04-28 17:54:51.655 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-04-28T17:54:51.662460997Z 2026-04-28 17:54:51.661 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-04-28T17:54:51.662474564Z 2026-04-28 17:54:51.662 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-04-28T17:54:51.662581291Z 2026-04-28 17:54:51.662 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-04-28T17:54:51.662721580Z 2026-04-28 17:54:51.662 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-28T17:54:51.662188439Z
2026-04-28T17:54:51.665290118Z 2026-04-28 17:54:51.664 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-04-28T17:54:51.685095825Z 2026-04-28 17:54:51.684 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-04-28T17:54:51.706494641Z 2026-04-28 17:54:51.706 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-28T17:54:51.832984656Z 2026-04-28 17:54:51.832 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-04-28T17:54:51.848855953Z 2026-04-28 17:54:51.848 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-28T17:54:51.848878235Z 2026-04-28 17:54:51.848 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-28T17:54:51.849031859Z 2026-04-28 17:54:51.848 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-28T17:54:51.957399469Z 2026-04-28 17:54:51.956 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-04-28T17:54:51.957417691Z 2026-04-28 17:54:51.957 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-04-28T17:54:51.957495297Z 2026-04-28 17:54:51.957 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-04-28T17:54:51.990253227Z 2026-04-28 17:54:51.989 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-04-28T17:54:52.687795445Z 2026-04-28 17:54:52.687 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-04-28T17:54:52.885366574Z 2026-04-28 17:54:52.882 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-04-28T17:54:52.886935169Z 2026-04-28 17:54:52.883 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-04-28T17:54:52.894331606Z 2026-04-28 17:54:52.893 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-28T17:54:52.900688016Z 2026-04-28 17:54:52.900 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-04-28T17:54:52.905258900Z 2026-04-28 17:54:52.904 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-28T17:54:52.905361191Z 2026-04-28 17:54:52.905 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-28T17:54:52.934102422Z 2026-04-28 17:54:52.933 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-04-28T17:54:52.942598338Z 2026-04-28 17:54:52.942 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-04-28T17:54:52.963269012Z 2026-04-28 17:54:52.962 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-28T17:54:52.976992831Z 2026-04-28 17:54:52.976 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-04-28T17:54:52.988530403Z 2026-04-28 17:54:52.988 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-04-28T17:54:52.998393244Z 2026-04-28 17:54:52.998 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-04-28T17:54:53.080773437Z 2026-04-28 17:54:53.080 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-04-28T17:54:53.361098302Z 2026-04-28 17:54:53.360 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-04-28T17:54:53.522003411Z 2026-04-28 17:54:53.521 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-04-28T17:54:53.780896023Z 2026-04-28 17:54:53.780 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-28T17:54:53.907226473Z 2026-04-28 17:54:53.906 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-04-28T17:54:53.913160578Z 2026-04-28 17:54:53.912 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_04_28.request.log
2026-04-28T17:54:53.917826509Z 2026-04-28 17:54:53.917 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-04-28T17:54:53.918565662Z 2026-04-28 17:54:53.918 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3714ms
2026-04-28T17:54:54.068140983Z 2026-04-28 17:54:54.067 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=40
2026-04-28T17:54:54.126399235Z 2026-04-28 17:54:54.126 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-04-28T17:54:57.204524371Z 2026-04-28 17:54:57.204 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-04-28T17:54:59.792293968Z 2026-04-28 17:54:59.791 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard1_replica_n2&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-04-28T17:54:59.836521444Z 2026-04-28 17:54:59.836 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-04-28T17:54:59.887355436Z 2026-04-28 17:54:59.886 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-04-28T17:55:00.166313370Z 2026-04-28 17:55:00.165 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-04-28T17:55:00.166415769Z 2026-04-28 17:55:00.166 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n2' using configuration from configset techproducts, trusted=true
2026-04-28T17:55:00.183681899Z 2026-04-28 17:55:00.183 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n2], dataDir=[/var/solr/data/test-collection_shard1_replica_n2/data/]
2026-04-28T17:55:00.593283057Z 2026-04-28 17:55:00.592 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-28T17:55:00.611285777Z 2026-04-28 17:55:00.610 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-04-28T17:55:00.614680702Z 2026-04-28 17:55:00.614 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-04-28T17:55:00.615674622Z 2026-04-28 17:55:00.615 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-04-28T17:55:00.615755398Z 2026-04-28 17:55:00.615 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-04-28T17:55:00.654479551Z 2026-04-28 17:55:00.653 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-04-28T17:55:00.656301643Z 2026-04-28 17:55:00.656 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-04-28T17:55:00.658656171Z 2026-04-28 17:55:00.658 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-04-28T17:55:00.660209018Z 2026-04-28 17:55:00.659 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-04-28T17:55:00.671213148Z 2026-04-28 17:55:00.670 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-04-28T17:55:00.678512364Z 2026-04-28 17:55:00.678 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-04-28T17:55:00.684771274Z 2026-04-28 17:55:00.683 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n2 solr-node1-6 core_node5 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-04-28T17:55:00.684787454Z 2026-04-28 17:55:00.683 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n2 solr-node1-6 core_node5 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-04-28T17:55:00.691288483Z 2026-04-28 17:55:00.690 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n2 solr-node1-6 core_node5 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-04-28T17:55:00.704589558Z 2026-04-28 17:55:00.703 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-04-28T17:55:00.705958181Z 2026-04-28 17:55:00.705 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-04-28T17:55:00.726257797Z 2026-04-28 17:55:00.725 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-04-28T17:55:00.726293649Z 2026-04-28 17:55:00.725 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-04-28T17:55:00.726388865Z 2026-04-28 17:55:00.726 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n2/
2026-04-28T17:55:00.729202466Z 2026-04-28 17:55:00.728 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-04-28T17:55:00.730480859Z 2026-04-28 17:55:00.730 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n2/ has no replicas
2026-04-28T17:55:00.731864370Z 2026-04-28 17:55:00.731 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72139557998690306-core_node5-n_0000000000
2026-04-28T17:55:00.862885215Z 2026-04-28 17:55:00.862 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n2/ shard1
2026-04-28T17:55:00.978474960Z 2026-04-28 17:55:00.977 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:00.983368132Z 2026-04-28 17:55:00.983 INFO (qtp985324122-59-solr-node1-6) [c:test-collection s:shard1 r:core_node5 x:test-collection_shard1_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-04-28T17:55:00.992528985Z 2026-04-28 17:55:00.992 INFO (qtp985324122-59-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1201
2026-04-28T17:55:01.093326956Z 2026-04-28 17:55:01.092 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:01.404112826Z 2026-04-28 17:55:01.403 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:01.514158880Z 2026-04-28 17:55:01.513 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2008971,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-04-28T17:54:49.99781681Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-04-28T17:54:50.386341569Z *** [WARN] *** Your open file limit is currently 1024.
2026-04-28T17:54:50.386373644Z It should be set to 65000 to avoid operational disruption.
2026-04-28T17:54:50.386376658Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-04-28T17:54:50.386673334Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-04-28T17:54:50.386683957Z
2026-04-28T17:54:50.395468188Z Java 17 detected. Enabled workaround for SOLR-16463
2026-04-28T17:54:50.402581668Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-04-28T17:54:50.449822192Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-04-28T17:54:50.478382065Z WARNING: A command line option has enabled the Security Manager
2026-04-28T17:54:50.478402893Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-04-28T17:54:55.608925031Z 2026-04-28 17:54:55.605 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-04-28T17:54:55.864057628Z 2026-04-28 17:54:55.863 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-04-28T17:54:55.888374677Z 2026-04-28 17:54:55.887 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-04-28T17:54:55.895203642Z 2026-04-28 17:54:55.894 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-04-28T17:54:55.895216883Z 2026-04-28 17:54:55.894 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-04-28T17:54:55.895287920Z 2026-04-28 17:54:55.894 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-04-28T17:54:55.895394350Z 2026-04-28 17:54:55.894 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-28T17:54:55.894897136Z
2026-04-28T17:54:55.897926123Z 2026-04-28 17:54:55.897 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-04-28T17:54:55.918948790Z 2026-04-28 17:54:55.918 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-04-28T17:54:55.941259042Z 2026-04-28 17:54:55.940 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-28T17:54:56.070252523Z 2026-04-28 17:54:56.069 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-04-28T17:54:56.085280132Z 2026-04-28 17:54:56.084 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-28T17:54:56.085298038Z 2026-04-28 17:54:56.084 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-28T17:54:56.085326017Z 2026-04-28 17:54:56.085 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-04-28T17:54:56.193510687Z 2026-04-28 17:54:56.192 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-04-28T17:54:56.193553749Z 2026-04-28 17:54:56.193 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-04-28T17:54:56.193654614Z 2026-04-28 17:54:56.193 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-04-28T17:54:56.225663405Z 2026-04-28 17:54:56.225 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-04-28T17:54:56.951262577Z 2026-04-28 17:54:56.950 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-04-28T17:54:57.079118206Z 2026-04-28 17:54:57.078 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-04-28T17:54:57.080222083Z 2026-04-28 17:54:57.078 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-04-28T17:54:57.088282613Z 2026-04-28 17:54:57.087 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-28T17:54:57.110346667Z 2026-04-28 17:54:57.108 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-04-28T17:54:57.111469802Z 2026-04-28 17:54:57.111 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-04-28T17:54:57.111583941Z 2026-04-28 17:54:57.111 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-04-28T17:54:57.144944327Z 2026-04-28 17:54:57.144 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-04-28T17:54:57.153784544Z 2026-04-28 17:54:57.153 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-04-28T17:54:57.176319086Z 2026-04-28 17:54:57.175 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-04-28T17:54:57.186274390Z 2026-04-28 17:54:57.185 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-04-28T17:54:57.195622946Z 2026-04-28 17:54:57.195 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-04-28T17:54:57.204414279Z 2026-04-28 17:54:57.204 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-04-28T17:54:57.247150155Z 2026-04-28 17:54:57.246 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-04-28T17:54:57.526409847Z 2026-04-28 17:54:57.525 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-04-28T17:54:57.683297508Z 2026-04-28 17:54:57.682 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-04-28T17:54:57.924773388Z 2026-04-28 17:54:57.924 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-28T17:54:58.049357565Z 2026-04-28 17:54:58.048 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-04-28T17:54:58.054348762Z 2026-04-28 17:54:58.054 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_04_28.request.log
2026-04-28T17:54:58.058077033Z 2026-04-28 17:54:58.057 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-04-28T17:54:58.059577501Z 2026-04-28 17:54:58.059 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @7658ms
2026-04-28T17:54:58.210085280Z 2026-04-28 17:54:58.209 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=40
2026-04-28T17:54:58.261559268Z 2026-04-28 17:54:58.261 INFO (qtp985324122-55-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-04-28T17:54:59.502889648Z 2026-04-28 17:54:59.502 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=4
2026-04-28T17:54:59.793303509Z 2026-04-28 17:54:59.792 INFO (qtp985324122-30-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-04-28T17:54:59.837184169Z 2026-04-28 17:54:59.836 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-04-28T17:54:59.887360431Z 2026-04-28 17:54:59.886 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-04-28T17:55:00.173583503Z 2026-04-28 17:55:00.172 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-04-28T17:55:00.173601786Z 2026-04-28 17:55:00.173 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n1' using configuration from configset techproducts, trusted=true
2026-04-28T17:55:00.190948135Z 2026-04-28 17:55:00.190 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n1], dataDir=[/var/solr/data/test-collection_shard2_replica_n1/data/]
2026-04-28T17:55:00.601426391Z 2026-04-28 17:55:00.601 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-04-28T17:55:00.619496657Z 2026-04-28 17:55:00.619 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-04-28T17:55:00.624529898Z 2026-04-28 17:55:00.624 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-04-28T17:55:00.625443024Z 2026-04-28 17:55:00.624 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-04-28T17:55:00.625512821Z 2026-04-28 17:55:00.625 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-04-28T17:55:00.664729753Z 2026-04-28 17:55:00.664 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-04-28T17:55:00.667447448Z 2026-04-28 17:55:00.667 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-04-28T17:55:00.669138214Z 2026-04-28 17:55:00.668 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-04-28T17:55:00.670623739Z 2026-04-28 17:55:00.670 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-04-28T17:55:00.682249072Z 2026-04-28 17:55:00.681 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-04-28T17:55:00.691363570Z 2026-04-28 17:55:00.689 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-04-28T17:55:00.697392181Z 2026-04-28 17:55:00.696 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-04-28T17:55:00.697409427Z 2026-04-28 17:55:00.696 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-04-28T17:55:00.703501169Z 2026-04-28 17:55:00.702 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n1 solr-node1-6 core_node3 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-04-28T17:55:00.712805415Z 2026-04-28 17:55:00.711 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-04-28T17:55:00.713043204Z 2026-04-28 17:55:00.712 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-04-28T17:55:00.859872320Z 2026-04-28 17:55:00.859 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-04-28T17:55:00.859903553Z 2026-04-28 17:55:00.859 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-04-28T17:55:00.860051900Z 2026-04-28 17:55:00.859 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n1/
2026-04-28T17:55:00.861195617Z 2026-04-28 17:55:00.860 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-04-28T17:55:00.862648342Z 2026-04-28 17:55:00.862 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ has no replicas
2026-04-28T17:55:00.863981235Z 2026-04-28 17:55:00.863 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72139557998690308-core_node3-n_0000000000
2026-04-28T17:55:00.872543165Z 2026-04-28 17:55:00.872 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n1/ shard2
2026-04-28T17:55:00.977606989Z 2026-04-28 17:55:00.977 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:00.983337810Z 2026-04-28 17:55:00.982 INFO (qtp985324122-30-solr-node1-6) [c:test-collection s:shard2 r:core_node3 x:test-collection_shard2_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-04-28T17:55:00.992230060Z 2026-04-28 17:55:00.991 INFO (qtp985324122-30-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1199
2026-04-28T17:55:01.093199144Z 2026-04-28 17:55:01.092 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:01.404119515Z 2026-04-28 17:55:01.403 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:01.425988896Z 2026-04-28 17:55:01.425 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-04-28T17:55:01.514169088Z 2026-04-28 17:55:01.513 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-04-28T17:55:01.516444155Z 2026-04-28 17:55:01.516 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2003
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2009154,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-04-28T17:54:50.191268503Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 83 Docker SolrCloud rolling upgrade # in 35634 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# 66b76b382631: Waiting
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Download complete
# af6eca94c810: Pull complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# cb0efb96dabd: Pull complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# f63eb04151bc: Already exists
# a1b8f7509ae1: Already exists
# e434fc527435: Already exists
# 3acf851d0cdf: Already exists
# 142ed9e09943: Already exists
# 923151477894: Pulling fs layer
# d68e9ab311bd: Pulling fs layer
# 867e89390420: Pulling fs layer
# df768b5159fa: Pulling fs layer
# 5e0136e7cd78: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# df768b5159fa: Waiting
# 5e0136e7cd78: Waiting
# 4f4fb700ef54: Waiting
# 867e89390420: Verifying Checksum
# 867e89390420: Download complete
# d68e9ab311bd: Download complete
# df768b5159fa: Download complete
# 5e0136e7cd78: Verifying Checksum
# 5e0136e7cd78: Download complete
# 4f4fb700ef54: Download complete
# 923151477894: Download complete
# 923151477894: Pull complete
# d68e9ab311bd: Pull complete
# 867e89390420: Pull complete
# df768b5159fa: Pull complete
# 5e0136e7cd78: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:880edcaeb8cd2b5688e539b3027585211531fcbdb17f304cd0e001cd90bb01aa
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 988f767d86c8b5dd3d835b0ebced50ed9b61989f69a8db2bf07344a07a667702
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# b3b27178d1981d52b63fdc1cf2f979af1fea47b357e066f0ccb95e7b0ad24af3
# Starting solr-node2 connected to first node's ZooKeeper
# a854db80fd01fef1a62c5a0ebceebce558602a42c6984a18e68869d988107385
# Started solr-node3 connected to first node's ZooKeeper
# 5f62111b727322731b303947f3f31407ead4d307ea978827dc59c7ac1b8c5d57
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-04-28 17:54:59.320; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-28 17:54:59.358; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 84 allowPaths - backup # in 14298 ms
ok 85 snapshot lifecycle # in 4497 ms
ok 86 snapshot list # in 3048 ms
ok 87 snapshot describe # in 3071 ms
ok 88 start solr with ssl # in 15139 ms
ok 89 use different hostname when not checking peer-name # in 23081 ms
ok 90 start solr with ssl and auth # in 13755 ms
ok 91 start solr with client truststore and security manager # in 15452 ms
ok 92 start solr with mTLS needed # in 43577 ms
ok 93 start solr with mTLS wanted # in 38489 ms
ok 94 test keystore reload # in 55420 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15356 ms
ok 96 stop command for single port # in 17618 ms
ok 97 check stop command doesn't hang # in 213 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7870 ms
ok 99 deprecated system properties converted to modern properties # in 195256 ms
ok 100 start with custom jetty options # in 7718 ms
ok 101 bootstrapping a configset # in 8746 ms
ok 102 status detects locally running solr # in 9128 ms
ok 103 status with --solr-url from user # in 7928 ms
ok 104 status with --port from user # in 7751 ms
ok 105 multiple connection options are prevented # in 790 ms
ok 106 status with invalid --solr-url from user # in 1228 ms
ok 107 status with --short format # in 195459 ms
ok 108 searching solr via locally executed streaming expression # in 1594 ms
ok 109 searching solr via remotely executed streaming expression # in 1183 ms
ok 110 variable interpolation # in 1545 ms
ok 111 searching solr without credentials fails with error # in 1054 ms
ok 112 --version returns Solr version # in 679 ms
ok 113 version as direct tool call still runs # in 673 ms
ok 114 short help # in 710 ms
ok 115 short help is inferred # in 726 ms
ok 116 long help # in 748 ms
ok 117 running subcommands with zk is prevented # in 681 ms
ok 118 listing out files # in 2055 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7322 ms
ok 120 copying files around # in 10396 ms
ok 121 upconfig # in 2443 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1039 ms
ok 123 downconfig # in 1086 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1347 ms
ok 125 env var ZK_HOST is honored # in 3037 ms
ok 126 Cleanup # in 1788 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 40s
128 actionable tasks: 128 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/ivkkggriai6mm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 192 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/192/
No tests ran.
Build Log:
[...truncated 1168 lines...]
# ERROR: Server refused connection at: http://localhost:43889/solr/test_args/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:43889/solr/test_args/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #10 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/79-10
# Last output:
# POSTing args to http://localhost:43889/solr/test_args/update...
# PostTool: WARNING: Connection error (is Solr running at http://localhost:43889/solr/test_args/update ?): java.net.ConnectException: Connection refused
# PostTool: WARNING: IOException while posting data: java.net.ConnectException: Connection refused
# PostTool: WARNING: IOException while reading response: java.net.ConnectException: Connection refused
# COMMITting Solr index changes to http://localhost:43889/solr/test_args/update...
#
# ERROR: Server refused connection at: http://localhost:43889/solr/test_args/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:43889/solr/test_args/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 80 stdin mode # in 1128 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 201)
# `assert_output --partial "Created collection 'test_stdin'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'test_stdin'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43889.
#
# ERROR: Server refused connection at: http://localhost:43889/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43889.
#
# ERROR: Server refused connection at: http://localhost:43889/solr/admin/info/system
not ok 81 verbose echo the Solr response # in 1110 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43889.
#
# ERROR: Server refused connection at: http://localhost:43889/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43889.
#
# ERROR: Server refused connection at: http://localhost:43889/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8902 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 15114 ms
ok 85 snapshot lifecycle # in 4592 ms
ok 86 snapshot list # in 3147 ms
ok 87 snapshot describe # in 3146 ms
ok 88 start solr with ssl # in 15816 ms
ok 89 use different hostname when not checking peer-name # in 23686 ms
ok 90 start solr with ssl and auth # in 14281 ms
ok 91 start solr with client truststore and security manager # in 16006 ms
ok 92 start solr with mTLS needed # in 46104 ms
ok 93 start solr with mTLS wanted # in 33153 ms
ok 94 test keystore reload # in 56791 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15632 ms
ok 96 stop command for single port # in 17734 ms
ok 97 check stop command doesn't hang # in 204 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7785 ms
ok 99 deprecated system properties converted to modern properties # in 194980 ms
ok 100 start with custom jetty options # in 7857 ms
ok 101 bootstrapping a configset # in 8854 ms
ok 102 status detects locally running solr # in 9274 ms
ok 103 status with --solr-url from user # in 7776 ms
ok 104 status with --port from user # in 7879 ms
ok 105 multiple connection options are prevented # in 816 ms
ok 106 status with invalid --solr-url from user # in 1196 ms
ok 107 status with --short format # in 196106 ms
ok 108 searching solr via locally executed streaming expression # in 1637 ms
ok 109 searching solr via remotely executed streaming expression # in 1184 ms
ok 110 variable interpolation # in 1584 ms
ok 111 searching solr without credentials fails with error # in 1077 ms
ok 112 --version returns Solr version # in 749 ms
ok 113 version as direct tool call still runs # in 690 ms
ok 114 short help # in 787 ms
ok 115 short help is inferred # in 751 ms
ok 116 long help # in 779 ms
ok 117 running subcommands with zk is prevented # in 726 ms
ok 118 listing out files # in 2087 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7475 ms
ok 120 copying files around # in 10691 ms
ok 121 upconfig # in 2461 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1159 ms
ok 123 downconfig # in 1151 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1374 ms
ok 125 env var ZK_HOST is honored # in 3083 ms
ok 126 Cleanup # in 1808 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 55s
129 actionable tasks: 129 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/debtj2qofaric
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 191 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/191/
No tests ran.
Build Log:
[...truncated 1167 lines...]
# ERROR: Server refused connection at: http://localhost:46569/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:46569/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# POSTing stdin to http://localhost:46569/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# COMMITting Solr index changes to http://localhost:46569/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:46569/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:46569/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 81 verbose echo the Solr response # in 1050 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46569.
#
# ERROR: Server refused connection at: http://localhost:46569/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46569.
#
# ERROR: Server refused connection at: http://localhost:46569/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8433 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 14428 ms
ok 85 snapshot lifecycle # in 4408 ms
ok 86 snapshot list # in 3053 ms
ok 87 snapshot describe # in 3129 ms
ok 88 start solr with ssl # in 15045 ms
ok 89 use different hostname when not checking peer-name # in 23311 ms
ok 90 start solr with ssl and auth # in 13676 ms
ok 91 start solr with client truststore and security manager # in 15386 ms
ok 92 start solr with mTLS needed # in 44090 ms
ok 93 start solr with mTLS wanted # in 32345 ms
ok 94 test keystore reload # in 55258 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15259 ms
ok 96 stop command for single port # in 17459 ms
ok 97 check stop command doesn't hang # in 204 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7649 ms
ok 99 deprecated system properties converted to modern properties # in 194981 ms
ok 100 start with custom jetty options # in 7623 ms
ok 101 bootstrapping a configset # in 8960 ms
ok 102 status detects locally running solr # in 9132 ms
ok 103 status with --solr-url from user # in 7708 ms
ok 104 status with --port from user # in 7777 ms
ok 105 multiple connection options are prevented # in 791 ms
ok 106 status with invalid --solr-url from user # in 1193 ms
ok 107 status with --short format # in 195739 ms
ok 108 searching solr via locally executed streaming expression # in 1590 ms
ok 109 searching solr via remotely executed streaming expression # in 1175 ms
ok 110 variable interpolation # in 1569 ms
ok 111 searching solr without credentials fails with error # in 1040 ms
ok 112 --version returns Solr version # in 680 ms
ok 113 version as direct tool call still runs # in 676 ms
ok 114 short help # in 736 ms
ok 115 short help is inferred # in 711 ms
ok 116 long help # in 707 ms
ok 117 running subcommands with zk is prevented # in 673 ms
ok 118 listing out files # in 2063 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7366 ms
ok 120 copying files around # in 10423 ms
ok 121 upconfig # in 2470 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1033 ms
ok 123 downconfig # in 1110 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1350 ms
ok 125 env var ZK_HOST is honored # in 3039 ms
ok 126 Cleanup # in 1818 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 25s
129 actionable tasks: 129 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/jwzejxjb475gs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 190 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/190/
No tests ran.
Build Log:
[...truncated 1176 lines...]
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:38419/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38419.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38419.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:38419/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c monitors_verbose -s http://localhost:38419 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-25 16:22:15.943; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-25 16:22:15.947; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:38419/solr/admin/collections
ok 82 post solr log into solr via cli # in 8835 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 14784 ms
ok 85 snapshot lifecycle # in 4743 ms
ok 86 snapshot list # in 3273 ms
ok 87 snapshot describe # in 3235 ms
ok 88 start solr with ssl # in 16047 ms
ok 89 use different hostname when not checking peer-name # in 24029 ms
ok 90 start solr with ssl and auth # in 14123 ms
ok 91 start solr with client truststore and security manager # in 16084 ms
ok 92 start solr with mTLS needed # in 45171 ms
ok 93 start solr with mTLS wanted # in 33843 ms
ok 94 test keystore reload # in 57416 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15630 ms
ok 96 stop command for single port # in 18030 ms
ok 97 check stop command doesn't hang # in 216 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8197 ms
ok 99 deprecated system properties converted to modern properties # in 195317 ms
ok 100 start with custom jetty options # in 8048 ms
ok 101 bootstrapping a configset # in 9178 ms
ok 102 status detects locally running solr # in 9313 ms
ok 103 status with --solr-url from user # in 7973 ms
ok 104 status with --port from user # in 7963 ms
ok 105 multiple connection options are prevented # in 827 ms
ok 106 status with invalid --solr-url from user # in 1207 ms
ok 107 status with --short format # in 195800 ms
ok 108 searching solr via locally executed streaming expression # in 1643 ms
ok 109 searching solr via remotely executed streaming expression # in 1256 ms
ok 110 variable interpolation # in 1602 ms
ok 111 searching solr without credentials fails with error # in 1101 ms
ok 112 --version returns Solr version # in 717 ms
ok 113 version as direct tool call still runs # in 733 ms
ok 114 short help # in 759 ms
ok 115 short help is inferred # in 802 ms
ok 116 long help # in 769 ms
ok 117 running subcommands with zk is prevented # in 704 ms
ok 118 listing out files # in 2119 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7636 ms
ok 120 copying files around # in 10790 ms
ok 121 upconfig # in 2491 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1163 ms
ok 123 downconfig # in 1155 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1442 ms
ok 125 env var ZK_HOST is honored # in 3149 ms
ok 126 Cleanup # in 1860 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 45s
129 actionable tasks: 129 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/3ezuviw4qiuiu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 189 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/189/
No tests ran.
Build Log:
[...truncated 1167 lines...]
# ERROR: Server refused connection at: http://localhost:35749/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:35749/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# POSTing stdin to http://localhost:35749/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# COMMITting Solr index changes to http://localhost:35749/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:35749/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:35749/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:486)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$0(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 81 verbose echo the Solr response # in 1065 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35749.
#
# ERROR: Server refused connection at: http://localhost:35749/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35749.
#
# ERROR: Server refused connection at: http://localhost:35749/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8545 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 14303 ms
ok 85 snapshot lifecycle # in 4463 ms
ok 86 snapshot list # in 3094 ms
ok 87 snapshot describe # in 3103 ms
ok 88 start solr with ssl # in 15220 ms
ok 89 use different hostname when not checking peer-name # in 23194 ms
ok 90 start solr with ssl and auth # in 13684 ms
ok 91 start solr with client truststore and security manager # in 15362 ms
ok 92 start solr with mTLS needed # in 44183 ms
ok 93 start solr with mTLS wanted # in 32656 ms
ok 94 test keystore reload # in 55751 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15289 ms
ok 96 stop command for single port # in 17517 ms
ok 97 check stop command doesn't hang # in 199 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7757 ms
ok 99 deprecated system properties converted to modern properties # in 195029 ms
ok 100 start with custom jetty options # in 7859 ms
ok 101 bootstrapping a configset # in 8716 ms
ok 102 status detects locally running solr # in 9003 ms
ok 103 status with --solr-url from user # in 7552 ms
ok 104 status with --port from user # in 7818 ms
ok 105 multiple connection options are prevented # in 795 ms
ok 106 status with invalid --solr-url from user # in 1186 ms
ok 107 status with --short format # in 195717 ms
ok 108 searching solr via locally executed streaming expression # in 1586 ms
ok 109 searching solr via remotely executed streaming expression # in 1173 ms
ok 110 variable interpolation # in 1557 ms
ok 111 searching solr without credentials fails with error # in 1050 ms
ok 112 --version returns Solr version # in 689 ms
ok 113 version as direct tool call still runs # in 678 ms
ok 114 short help # in 716 ms
ok 115 short help is inferred # in 713 ms
ok 116 long help # in 716 ms
ok 117 running subcommands with zk is prevented # in 677 ms
ok 118 listing out files # in 2071 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7337 ms
ok 120 copying files around # in 10412 ms
ok 121 upconfig # in 2451 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1042 ms
ok 123 downconfig # in 1101 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1333 ms
ok 125 env var ZK_HOST is honored # in 3043 ms
ok 126 Cleanup # in 1801 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 27m 52s
129 actionable tasks: 129 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/6d6eka7ascrqs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 188 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/188/
No tests ran.
Build Log:
[...truncated 1165 lines...]
# ERROR: Server refused connection at: http://localhost:41807/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:41807/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Posting files to [base] url http://localhost:41807/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:41807/solr/monitors_verbose/update...
#
# ERROR: Server refused connection at: http://localhost:41807/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:41807/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:506)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 82 post solr log into solr via cli # in 8691 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 14788 ms
ok 85 snapshot lifecycle # in 4581 ms
ok 86 snapshot list # in 3145 ms
ok 87 snapshot describe # in 3225 ms
ok 88 start solr with ssl # in 15501 ms
ok 89 use different hostname when not checking peer-name # in 23805 ms
ok 90 start solr with ssl and auth # in 14261 ms
ok 91 start solr with client truststore and security manager # in 15838 ms
ok 92 start solr with mTLS needed # in 45260 ms
ok 93 start solr with mTLS wanted # in 34633 ms
ok 94 test keystore reload # in 57091 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15554 ms
ok 96 stop command for single port # in 17888 ms
ok 97 check stop command doesn't hang # in 212 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8158 ms
ok 99 deprecated system properties converted to modern properties # in 194879 ms
ok 100 start with custom jetty options # in 7883 ms
ok 101 bootstrapping a configset # in 9125 ms
ok 102 status detects locally running solr # in 9315 ms
ok 103 status with --solr-url from user # in 7894 ms
ok 104 status with --port from user # in 7897 ms
ok 105 multiple connection options are prevented # in 818 ms
ok 106 status with invalid --solr-url from user # in 1195 ms
ok 107 status with --short format # in 195585 ms
ok 108 searching solr via locally executed streaming expression # in 1640 ms
ok 109 searching solr via remotely executed streaming expression # in 1242 ms
ok 110 variable interpolation # in 1591 ms
ok 111 searching solr without credentials fails with error # in 1078 ms
ok 112 --version returns Solr version # in 716 ms
ok 113 version as direct tool call still runs # in 714 ms
ok 114 short help # in 816 ms
ok 115 short help is inferred # in 729 ms
ok 116 long help # in 765 ms
ok 117 running subcommands with zk is prevented # in 701 ms
ok 118 listing out files # in 2118 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7492 ms
ok 120 copying files around # in 10664 ms
ok 121 upconfig # in 2531 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1151 ms
ok 123 downconfig # in 1158 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1440 ms
ok 125 env var ZK_HOST is honored # in 3133 ms
ok 126 Cleanup # in 1817 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 32s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/v3u5zz25373tq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 187 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/187/
No tests ran.
Build Log:
[...truncated 1170 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:35129/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35129.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35129.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:35129/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:35129 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-22 12:40:38.860; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-22 12:40:38.864; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:35129/solr/admin/collections
not ok 81 verbose echo the Solr response # in 1120 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35129.
#
# ERROR: Server refused connection at: http://localhost:35129/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35129.
#
# ERROR: Server refused connection at: http://localhost:35129/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8789 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 15107 ms
ok 85 snapshot lifecycle # in 4897 ms
ok 86 snapshot list # in 3289 ms
ok 87 snapshot describe # in 3299 ms
ok 88 start solr with ssl # in 15903 ms
ok 89 use different hostname when not checking peer-name # in 24222 ms
ok 90 start solr with ssl and auth # in 14626 ms
ok 91 start solr with client truststore and security manager # in 16165 ms
ok 92 start solr with mTLS needed # in 46487 ms
ok 93 start solr with mTLS wanted # in 35474 ms
ok 94 test keystore reload # in 57401 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15711 ms
ok 96 stop command for single port # in 17971 ms
ok 97 check stop command doesn't hang # in 206 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8013 ms
ok 99 deprecated system properties converted to modern properties # in 195431 ms
ok 100 start with custom jetty options # in 7820 ms
ok 101 bootstrapping a configset # in 8853 ms
ok 102 status detects locally running solr # in 9236 ms
ok 103 status with --solr-url from user # in 7692 ms
ok 104 status with --port from user # in 7759 ms
ok 105 multiple connection options are prevented # in 808 ms
ok 106 status with invalid --solr-url from user # in 1208 ms
ok 107 status with --short format # in 195573 ms
ok 108 searching solr via locally executed streaming expression # in 1658 ms
ok 109 searching solr via remotely executed streaming expression # in 1227 ms
ok 110 variable interpolation # in 1632 ms
ok 111 searching solr without credentials fails with error # in 1138 ms
ok 112 --version returns Solr version # in 696 ms
ok 113 version as direct tool call still runs # in 702 ms
ok 114 short help # in 778 ms
ok 115 short help is inferred # in 741 ms
ok 116 long help # in 772 ms
ok 117 running subcommands with zk is prevented # in 699 ms
ok 118 listing out files # in 2121 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7545 ms
ok 120 copying files around # in 10584 ms
ok 121 upconfig # in 2514 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1101 ms
ok 123 downconfig # in 1134 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1373 ms
ok 125 env var ZK_HOST is honored # in 3103 ms
ok 126 Cleanup # in 1860 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 9s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/gdpy5smc63pze
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 186 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/186/
No tests ran.
Build Log:
[...truncated 1169 lines...]
# ERROR: Failed to create collection 'test_args' due to: IOException occurred when talking to server at: http://localhost:45831/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #10 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/79-10
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45831.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45831.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:45831/solr/test_args/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_args -s http://localhost:45831 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-21 13:05:47.105; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-21 13:05:47.108; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_args' due to: IOException occurred when talking to server at: http://localhost:45831/solr/admin/collections
not ok 80 stdin mode # in 1157 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 201)
# `assert_output --partial "Created collection 'test_stdin'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'test_stdin'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45831.
#
# ERROR: Server refused connection at: http://localhost:45831/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45831.
#
# ERROR: Server refused connection at: http://localhost:45831/solr/admin/info/system
not ok 81 verbose echo the Solr response # in 1053 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45831.
#
# ERROR: Server refused connection at: http://localhost:45831/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45831.
#
# ERROR: Server refused connection at: http://localhost:45831/solr/admin/info/system
ok 82 post solr log into solr via cli # in 8918 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 14388 ms
ok 85 snapshot lifecycle # in 4508 ms
ok 86 snapshot list # in 3161 ms
ok 87 snapshot describe # in 3135 ms
ok 88 start solr with ssl # in 15158 ms
ok 89 use different hostname when not checking peer-name # in 23367 ms
ok 90 start solr with ssl and auth # in 13886 ms
ok 91 start solr with client truststore and security manager # in 15535 ms
ok 92 start solr with mTLS needed # in 43965 ms
ok 93 start solr with mTLS wanted # in 32670 ms
ok 94 test keystore reload # in 56033 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15408 ms
ok 96 stop command for single port # in 17567 ms
ok 97 check stop command doesn't hang # in 201 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7705 ms
ok 99 deprecated system properties converted to modern properties # in 195240 ms
ok 100 start with custom jetty options # in 7744 ms
ok 101 bootstrapping a configset # in 8930 ms
ok 102 status detects locally running solr # in 9065 ms
ok 103 status with --solr-url from user # in 7770 ms
ok 104 status with --port from user # in 7617 ms
ok 105 multiple connection options are prevented # in 803 ms
ok 106 status with invalid --solr-url from user # in 1171 ms
ok 107 status with --short format # in 195954 ms
ok 108 searching solr via locally executed streaming expression # in 1617 ms
ok 109 searching solr via remotely executed streaming expression # in 1194 ms
ok 110 variable interpolation # in 1567 ms
ok 111 searching solr without credentials fails with error # in 1067 ms
ok 112 --version returns Solr version # in 686 ms
ok 113 version as direct tool call still runs # in 685 ms
ok 114 short help # in 713 ms
ok 115 short help is inferred # in 742 ms
ok 116 long help # in 730 ms
ok 117 running subcommands with zk is prevented # in 676 ms
ok 118 listing out files # in 2130 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7423 ms
ok 120 copying files around # in 10539 ms
ok 121 upconfig # in 2467 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1062 ms
ok 123 downconfig # in 1132 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1374 ms
ok 125 env var ZK_HOST is honored # in 3082 ms
ok 126 Cleanup # in 1814 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 27m 58s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/y2xfxv7v755qq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 185 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/185/
No tests ran.
Build Log:
[...truncated 1171 lines...]
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:45693/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45693.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:45693.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:45693/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c monitors_verbose -s http://localhost:45693 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-20 12:23:33.573; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-20 12:23:33.576; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:45693/solr/admin/collections
ok 82 post solr log into solr via cli # in 8934 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 15034 ms
ok 85 snapshot lifecycle # in 4682 ms
ok 86 snapshot list # in 3212 ms
ok 87 snapshot describe # in 3247 ms
ok 88 start solr with ssl # in 15683 ms
ok 89 use different hostname when not checking peer-name # in 24345 ms
ok 90 start solr with ssl and auth # in 14227 ms
ok 91 start solr with client truststore and security manager # in 15981 ms
ok 92 start solr with mTLS needed # in 45497 ms
ok 93 start solr with mTLS wanted # in 33852 ms
ok 94 test keystore reload # in 58227 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15686 ms
ok 96 stop command for single port # in 17918 ms
ok 97 check stop command doesn't hang # in 212 ms
not ok 98 SOLR-16976 solr starts with remote JMX enabled # in 197899 ms
# (from function `refute_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 360,
# in test file test/test_start_solr.bats, line 81)
# `refute_output --partial 'Exception'' failed
# Waiting up to 180 seconds to see Solr running on port 45693 [|] [/] [-] [\] [|]
# Started Solr server on port 45693 (pid=1795415). Happy searching!
#
#
# -- output should not contain substring --
# substring (1 lines):
# Exception
# output (14 lines):
# [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
# WARNING: Using incubator modules: jdk.incubator.vector
# WARNING: A command line option has enabled the Security Manager
# WARNING: The Security Manager is deprecated and will be removed in a future release
# Error binding ShutdownMonitor to port 44693: java.net.BindException: Address already in use
# 2026-04-20 12:28:31.308 INFO (main) [] o.e.j.s.Server jetty-12.0.27; built: 2025-09-10T23:47:49.595Z; git: 3569a3e83ad136ee44e26b370b74c1c5e9f33e61; jvm 21.0.4+7-LTS
# 2026-04-20 12:28:31.761 INFO (main) [] o.e.j.s.h.ContextHandler Started oeje10w.WebAppContext@6d467c87{solr-jetty-context,/solr,b=file:///home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server/solr-webapp/webapp/,a=AVAILABLE,h=oeje10s.SessionHandler@29182679{STARTED}}{/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server/solr-webapp/webapp}
# 2026-04-20 12:28:31.771 INFO (main) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
# 2026-04-20 12:28:31.778 INFO (main) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 10.1.0-SNAPSHOT b6bd7d5961fb86fe9a9d64ad759c38b4700dd899 [snapshot build, details omitted]
# 2026-04-20 12:28:31.778 INFO (main) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 45693
# 2026-04-20 12:28:31.779 INFO (main) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT
# 2026-04-20 12:28:31.779 INFO (main) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-20T12:28:31.779079887Z
# 2026-04-20 12:28:31.781 INFO (main) [] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/jvm_crash_1795415.log
# [2.509s][warning][jfr,system] Could not initialize JDK events. access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.event")
# --
#
# Please find the SOLR_HOME snapshot for failed test #4 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/98-4
# Last output:
# [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
# WARNING: Using incubator modules: jdk.incubator.vector
# WARNING: A command line option has enabled the Security Manager
# WARNING: The Security Manager is deprecated and will be removed in a future release
# Error binding ShutdownMonitor to port 44693: java.net.BindException: Address already in use
# 2026-04-20 12:28:31.308 INFO (main) [] o.e.j.s.Server jetty-12.0.27; built: 2025-09-10T23:47:49.595Z; git: 3569a3e83ad136ee44e26b370b74c1c5e9f33e61; jvm 21.0.4+7-LTS
# 2026-04-20 12:28:31.761 INFO (main) [] o.e.j.s.h.ContextHandler Started oeje10w.WebAppContext@6d467c87{solr-jetty-context,/solr,b=file:///home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server/solr-webapp/webapp/,a=AVAILABLE,h=oeje10s.SessionHandler@29182679{STARTED}}{/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT/server/solr-webapp/webapp}
# 2026-04-20 12:28:31.771 INFO (main) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
# 2026-04-20 12:28:31.778 INFO (main) [] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 10.1.0-SNAPSHOT b6bd7d5961fb86fe9a9d64ad759c38b4700dd899 [snapshot build, details omitted]
# 2026-04-20 12:28:31.778 INFO (main) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 45693
# 2026-04-20 12:28:31.779 INFO (main) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/solr-10.1.0-SNAPSHOT
# 2026-04-20 12:28:31.779 INFO (main) [] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-04-20T12:28:31.779079887Z
# 2026-04-20 12:28:31.781 INFO (main) [] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/solr-home/logs/jvm_crash_1795415.log
# [2.509s][warning][jfr,system] Could not initialize JDK events. access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.event")
ok 99 deprecated system properties converted to modern properties # in 194958 ms
ok 100 start with custom jetty options # in 8090 ms
ok 101 bootstrapping a configset # in 8911 ms
ok 102 status detects locally running solr # in 9224 ms
ok 103 status with --solr-url from user # in 7869 ms
ok 104 status with --port from user # in 7963 ms
ok 105 multiple connection options are prevented # in 836 ms
ok 106 status with invalid --solr-url from user # in 1170 ms
ok 107 status with --short format # in 195723 ms
ok 108 searching solr via locally executed streaming expression # in 1665 ms
ok 109 searching solr via remotely executed streaming expression # in 1226 ms
ok 110 variable interpolation # in 1672 ms
ok 111 searching solr without credentials fails with error # in 1076 ms
ok 112 --version returns Solr version # in 709 ms
ok 113 version as direct tool call still runs # in 732 ms
ok 114 short help # in 738 ms
ok 115 short help is inferred # in 729 ms
ok 116 long help # in 753 ms
ok 117 running subcommands with zk is prevented # in 718 ms
ok 118 listing out files # in 2143 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7694 ms
ok 120 copying files around # in 10782 ms
ok 121 upconfig # in 2482 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1218 ms
ok 123 downconfig # in 1183 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1379 ms
ok 125 env var ZK_HOST is honored # in 3083 ms
ok 126 Cleanup # in 1836 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 31m 46s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/d3b2w243vj33u
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 184 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/184/
No tests ran.
Build Log:
[...truncated 1165 lines...]
# ERROR: Server refused connection at: http://localhost:37341/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:37341/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12
# Last output:
# Posting files to [base] url http://localhost:37341/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:37341/solr/monitors_verbose/update...
#
# ERROR: Server refused connection at: http://localhost:37341/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:37341/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 82 post solr log into solr via cli # in 8465 ms
# Docker artifacts saved to:
ok 83 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 84 allowPaths - backup # in 14691 ms
ok 85 snapshot lifecycle # in 4567 ms
ok 86 snapshot list # in 3148 ms
ok 87 snapshot describe # in 3154 ms
ok 88 start solr with ssl # in 15257 ms
ok 89 use different hostname when not checking peer-name # in 23755 ms
ok 90 start solr with ssl and auth # in 13957 ms
ok 91 start solr with client truststore and security manager # in 15591 ms
ok 92 start solr with mTLS needed # in 43994 ms
ok 93 start solr with mTLS wanted # in 33123 ms
ok 94 test keystore reload # in 56383 ms
ok 95 SOLR-11740 check 'solr stop' connection # in 15332 ms
ok 96 stop command for single port # in 17630 ms
ok 97 check stop command doesn't hang # in 202 ms
ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7638 ms
ok 99 deprecated system properties converted to modern properties # in 195215 ms
ok 100 start with custom jetty options # in 7773 ms
ok 101 bootstrapping a configset # in 9005 ms
ok 102 status detects locally running solr # in 9054 ms
ok 103 status with --solr-url from user # in 7685 ms
ok 104 status with --port from user # in 7710 ms
ok 105 multiple connection options are prevented # in 793 ms
ok 106 status with invalid --solr-url from user # in 1163 ms
ok 107 status with --short format # in 195973 ms
ok 108 searching solr via locally executed streaming expression # in 1633 ms
ok 109 searching solr via remotely executed streaming expression # in 1163 ms
ok 110 variable interpolation # in 1572 ms
ok 111 searching solr without credentials fails with error # in 1059 ms
ok 112 --version returns Solr version # in 683 ms
ok 113 version as direct tool call still runs # in 699 ms
ok 114 short help # in 717 ms
ok 115 short help is inferred # in 730 ms
ok 116 long help # in 766 ms
ok 117 running subcommands with zk is prevented # in 696 ms
ok 118 listing out files # in 2067 ms
ok 119 connecting to solr via various solr urls and zk hosts # in 7523 ms
ok 120 copying files around # in 10499 ms
ok 121 upconfig # in 2477 ms
ok 122 SOLR-12429 test upconfig fails with symlink # in 1130 ms
ok 123 downconfig # in 1120 ms
ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1359 ms
ok 125 env var ZK_HOST is honored # in 3063 ms
ok 126 Cleanup # in 1800 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 30s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/tpzphqa6w3f2g
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 183 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/183/
No tests ran.
Build Log:
[...truncated 1211 lines...]
BUILD FAILED in 28m 31s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/y5qf7gg25ho4e
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 182 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/182/
No tests ran.
Build Log:
[...truncated 1162 lines...]
# ERROR: Server refused connection at: http://localhost:39599/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:39599/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Posting files to [base] url http://localhost:39599/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:39599/solr/monitors_verbose/update...
#
# ERROR: Server refused connection at: http://localhost:39599/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:39599/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 79 post solr log into solr via cli # in 8735 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14453 ms
ok 82 snapshot lifecycle # in 4445 ms
ok 83 snapshot list # in 3092 ms
ok 84 snapshot describe # in 3083 ms
ok 85 start solr with ssl # in 15118 ms
ok 86 use different hostname when not checking peer-name # in 23051 ms
ok 87 start solr with ssl and auth # in 13694 ms
ok 88 start solr with client truststore and security manager # in 15548 ms
ok 89 start solr with mTLS needed # in 47371 ms
ok 90 start solr with mTLS wanted # in 33709 ms
ok 91 test keystore reload # in 55907 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15394 ms
ok 93 stop command for single port # in 17597 ms
ok 94 check stop command doesn't hang # in 199 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7738 ms
ok 96 deprecated system properties converted to modern properties # in 195186 ms
ok 97 start with custom jetty options # in 8027 ms
ok 98 bootstrapping a configset # in 8786 ms
ok 99 status detects locally running solr # in 9247 ms
ok 100 status with --solr-url from user # in 7603 ms
ok 101 status with --port from user # in 7705 ms
ok 102 multiple connection options are prevented # in 799 ms
ok 103 status with invalid --solr-url from user # in 1136 ms
ok 104 status with --short format # in 195991 ms
ok 105 searching solr via locally executed streaming expression # in 1610 ms
ok 106 searching solr via remotely executed streaming expression # in 1187 ms
ok 107 variable interpolation # in 1582 ms
ok 108 searching solr without credentials fails with error # in 1064 ms
ok 109 --version returns Solr version # in 685 ms
ok 110 version as direct tool call still runs # in 694 ms
ok 111 short help # in 735 ms
ok 112 short help is inferred # in 742 ms
ok 113 long help # in 723 ms
ok 114 running subcommands with zk is prevented # in 721 ms
ok 115 listing out files # in 2093 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7429 ms
ok 117 copying files around # in 10550 ms
ok 118 upconfig # in 2454 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1048 ms
ok 120 downconfig # in 1119 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1366 ms
ok 122 env var ZK_HOST is honored # in 3086 ms
ok 123 Cleanup # in 1813 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 39s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/o76fjcvzi4nqg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 181 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/181/
No tests ran.
Build Log:
[...truncated 1158 lines...]
# ERROR: Server refused connection at: http://localhost:46069/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46069.
#
# ERROR: Server refused connection at: http://localhost:46069/solr/admin/info/system
ok 79 post solr log into solr via cli # in 8505 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14527 ms
ok 82 snapshot lifecycle # in 4509 ms
ok 83 snapshot list # in 3110 ms
ok 84 snapshot describe # in 3099 ms
ok 85 start solr with ssl # in 15153 ms
ok 86 use different hostname when not checking peer-name # in 23515 ms
ok 87 start solr with ssl and auth # in 13883 ms
ok 88 start solr with client truststore and security manager # in 15369 ms
ok 89 start solr with mTLS needed # in 44499 ms
ok 90 start solr with mTLS wanted # in 32832 ms
ok 91 test keystore reload # in 56486 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15289 ms
ok 93 stop command for single port # in 17447 ms
ok 94 check stop command doesn't hang # in 203 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7695 ms
ok 96 deprecated system properties converted to modern properties # in 195147 ms
ok 97 start with custom jetty options # in 7639 ms
ok 98 bootstrapping a configset # in 8728 ms
ok 99 status detects locally running solr # in 9213 ms
ok 100 status with --solr-url from user # in 7636 ms
ok 101 status with --port from user # in 7697 ms
ok 102 multiple connection options are prevented # in 794 ms
ok 103 status with invalid --solr-url from user # in 1195 ms
ok 104 status with --short format # in 196024 ms
ok 105 searching solr via locally executed streaming expression # in 1621 ms
ok 106 searching solr via remotely executed streaming expression # in 1194 ms
ok 107 variable interpolation # in 1569 ms
ok 108 searching solr without credentials fails with error # in 1058 ms
ok 109 --version returns Solr version # in 687 ms
ok 110 version as direct tool call still runs # in 712 ms
ok 111 short help # in 712 ms
ok 112 short help is inferred # in 739 ms
ok 113 long help # in 727 ms
ok 114 running subcommands with zk is prevented # in 690 ms
ok 115 listing out files # in 2078 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7417 ms
ok 117 copying files around # in 10570 ms
ok 118 upconfig # in 2464 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1289 ms
ok 120 downconfig # in 1138 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1357 ms
ok 122 env var ZK_HOST is honored # in 3125 ms
ok 123 Cleanup # in 1800 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 27m 45s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/imhxo62dgg3fo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 180 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/180/
No tests ran.
Build Log:
[...truncated 1159 lines...]
# ERROR: Server refused connection at: http://localhost:36215/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:36215/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# POSTing stdin to http://localhost:36215/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# COMMITting Solr index changes to http://localhost:36215/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:36215/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:36215/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 78 verbose echo the Solr response # in 1054 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:36215.
#
# ERROR: Server refused connection at: http://localhost:36215/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:36215.
#
# ERROR: Server refused connection at: http://localhost:36215/solr/admin/info/system
ok 79 post solr log into solr via cli # in 8483 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14363 ms
ok 82 snapshot lifecycle # in 4469 ms
ok 83 snapshot list # in 3104 ms
ok 84 snapshot describe # in 3119 ms
ok 85 start solr with ssl # in 15120 ms
ok 86 use different hostname when not checking peer-name # in 23162 ms
ok 87 start solr with ssl and auth # in 13656 ms
ok 88 start solr with client truststore and security manager # in 15460 ms
ok 89 start solr with mTLS needed # in 44175 ms
ok 90 start solr with mTLS wanted # in 33581 ms
ok 91 test keystore reload # in 58308 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15364 ms
ok 93 stop command for single port # in 17470 ms
ok 94 check stop command doesn't hang # in 202 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7716 ms
ok 96 deprecated system properties converted to modern properties # in 195000 ms
ok 97 start with custom jetty options # in 7694 ms
ok 98 bootstrapping a configset # in 8783 ms
ok 99 status detects locally running solr # in 8941 ms
ok 100 status with --solr-url from user # in 7636 ms
ok 101 status with --port from user # in 7633 ms
ok 102 multiple connection options are prevented # in 787 ms
ok 103 status with invalid --solr-url from user # in 1196 ms
ok 104 status with --short format # in 195672 ms
ok 105 searching solr via locally executed streaming expression # in 1594 ms
ok 106 searching solr via remotely executed streaming expression # in 1178 ms
ok 107 variable interpolation # in 1569 ms
ok 108 searching solr without credentials fails with error # in 1058 ms
ok 109 --version returns Solr version # in 674 ms
ok 110 version as direct tool call still runs # in 675 ms
ok 111 short help # in 705 ms
ok 112 short help is inferred # in 722 ms
ok 113 long help # in 715 ms
ok 114 running subcommands with zk is prevented # in 675 ms
ok 115 listing out files # in 2065 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7427 ms
ok 117 copying files around # in 10438 ms
ok 118 upconfig # in 2452 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1053 ms
ok 120 downconfig # in 1124 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1379 ms
ok 122 env var ZK_HOST is honored # in 3033 ms
ok 123 Cleanup # in 1802 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 30s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/djckepbdvafrc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 179 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/179/
No tests ran.
Build Log:
[...truncated 1167 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:44637/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44637.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44637.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:44637/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:44637 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-14 17:41:19.769; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-14 17:41:19.772; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:44637/solr/admin/collections
not ok 78 verbose echo the Solr response # in 1047 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44637.
#
# ERROR: Server refused connection at: http://localhost:44637/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44637.
#
# ERROR: Server refused connection at: http://localhost:44637/solr/admin/info/system
ok 79 post solr log into solr via cli # in 8440 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14415 ms
ok 82 snapshot lifecycle # in 4453 ms
ok 83 snapshot list # in 3133 ms
ok 84 snapshot describe # in 3064 ms
ok 85 start solr with ssl # in 15033 ms
ok 86 use different hostname when not checking peer-name # in 23366 ms
ok 87 start solr with ssl and auth # in 13811 ms
ok 88 start solr with client truststore and security manager # in 15525 ms
ok 89 start solr with mTLS needed # in 43796 ms
ok 90 start solr with mTLS wanted # in 32441 ms
ok 91 test keystore reload # in 55974 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15308 ms
ok 93 stop command for single port # in 17439 ms
ok 94 check stop command doesn't hang # in 196 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7645 ms
ok 96 deprecated system properties converted to modern properties # in 194928 ms
ok 97 start with custom jetty options # in 7704 ms
ok 98 bootstrapping a configset # in 8808 ms
ok 99 status detects locally running solr # in 8943 ms
ok 100 status with --solr-url from user # in 7659 ms
ok 101 status with --port from user # in 7632 ms
ok 102 multiple connection options are prevented # in 797 ms
ok 103 status with invalid --solr-url from user # in 1311 ms
ok 104 status with --short format # in 195666 ms
ok 105 searching solr via locally executed streaming expression # in 1622 ms
ok 106 searching solr via remotely executed streaming expression # in 1195 ms
ok 107 variable interpolation # in 1566 ms
ok 108 searching solr without credentials fails with error # in 1045 ms
ok 109 --version returns Solr version # in 681 ms
ok 110 version as direct tool call still runs # in 683 ms
ok 111 short help # in 707 ms
ok 112 short help is inferred # in 715 ms
ok 113 long help # in 738 ms
ok 114 running subcommands with zk is prevented # in 677 ms
ok 115 listing out files # in 2051 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7419 ms
ok 117 copying files around # in 10447 ms
ok 118 upconfig # in 2457 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1034 ms
ok 120 downconfig # in 1110 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1355 ms
ok 122 env var ZK_HOST is honored # in 3027 ms
ok 123 Cleanup # in 1806 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 30s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vz2sj3fhciidq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 178 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/178/
No tests ran.
Build Log:
[...truncated 1162 lines...]
# ERROR: Server refused connection at: http://localhost:41589/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:41589/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Posting files to [base] url http://localhost:41589/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:41589/solr/monitors_verbose/update...
#
# ERROR: Server refused connection at: http://localhost:41589/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:41589/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 79 post solr log into solr via cli # in 9446 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 15569 ms
ok 82 snapshot lifecycle # in 4826 ms
ok 83 snapshot list # in 3375 ms
ok 84 snapshot describe # in 3564 ms
ok 85 start solr with ssl # in 16886 ms
ok 86 use different hostname when not checking peer-name # in 25935 ms
ok 87 start solr with ssl and auth # in 15059 ms
ok 88 start solr with client truststore and security manager # in 17003 ms
ok 89 start solr with mTLS needed # in 45650 ms
ok 90 start solr with mTLS wanted # in 35042 ms
ok 91 test keystore reload # in 59046 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 16021 ms
ok 93 stop command for single port # in 18416 ms
ok 94 check stop command doesn't hang # in 219 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 8249 ms
ok 96 deprecated system properties converted to modern properties # in 195068 ms
ok 97 start with custom jetty options # in 8258 ms
ok 98 bootstrapping a configset # in 9756 ms
ok 99 status detects locally running solr # in 9724 ms
ok 100 status with --solr-url from user # in 8451 ms
ok 101 status with --port from user # in 8216 ms
ok 102 multiple connection options are prevented # in 927 ms
ok 103 status with invalid --solr-url from user # in 1310 ms
ok 104 status with --short format # in 195915 ms
ok 105 searching solr via locally executed streaming expression # in 1677 ms
ok 106 searching solr via remotely executed streaming expression # in 1320 ms
ok 107 variable interpolation # in 1697 ms
ok 108 searching solr without credentials fails with error # in 1134 ms
ok 109 --version returns Solr version # in 727 ms
ok 110 version as direct tool call still runs # in 770 ms
ok 111 short help # in 810 ms
ok 112 short help is inferred # in 805 ms
ok 113 long help # in 792 ms
ok 114 running subcommands with zk is prevented # in 711 ms
ok 115 listing out files # in 2165 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7850 ms
ok 117 copying files around # in 11119 ms
ok 118 upconfig # in 2629 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1118 ms
ok 120 downconfig # in 1217 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1412 ms
ok 122 env var ZK_HOST is honored # in 3171 ms
ok 123 Cleanup # in 1861 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 28s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/mquxwuelnuz4o
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 177 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/177/
No tests ran.
Build Log:
[...truncated 1159 lines...]
# ERROR: Server refused connection at: http://localhost:41957/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:41957/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# POSTing stdin to http://localhost:41957/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# COMMITting Solr index changes to http://localhost:41957/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:41957/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:41957/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 78 verbose echo the Solr response # in 1043 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:41957.
#
# ERROR: Server refused connection at: http://localhost:41957/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:41957.
#
# ERROR: Server refused connection at: http://localhost:41957/solr/admin/info/system
ok 79 post solr log into solr via cli # in 8523 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14352 ms
ok 82 snapshot lifecycle # in 4420 ms
ok 83 snapshot list # in 3118 ms
ok 84 snapshot describe # in 3096 ms
ok 85 start solr with ssl # in 15264 ms
ok 86 use different hostname when not checking peer-name # in 23500 ms
ok 87 start solr with ssl and auth # in 13744 ms
ok 88 start solr with client truststore and security manager # in 15389 ms
ok 89 start solr with mTLS needed # in 43964 ms
ok 90 start solr with mTLS wanted # in 33600 ms
ok 91 test keystore reload # in 55736 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15341 ms
ok 93 stop command for single port # in 17443 ms
ok 94 check stop command doesn't hang # in 199 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7740 ms
ok 96 deprecated system properties converted to modern properties # in 194985 ms
ok 97 start with custom jetty options # in 7814 ms
ok 98 bootstrapping a configset # in 8626 ms
ok 99 status detects locally running solr # in 8925 ms
ok 100 status with --solr-url from user # in 7572 ms
ok 101 status with --port from user # in 7630 ms
ok 102 multiple connection options are prevented # in 786 ms
ok 103 status with invalid --solr-url from user # in 1150 ms
ok 104 status with --short format # in 195723 ms
ok 105 searching solr via locally executed streaming expression # in 1628 ms
ok 106 searching solr via remotely executed streaming expression # in 1181 ms
ok 107 variable interpolation # in 1572 ms
ok 108 searching solr without credentials fails with error # in 1052 ms
ok 109 --version returns Solr version # in 693 ms
ok 110 version as direct tool call still runs # in 696 ms
ok 111 short help # in 718 ms
ok 112 short help is inferred # in 735 ms
ok 113 long help # in 725 ms
ok 114 running subcommands with zk is prevented # in 692 ms
ok 115 listing out files # in 2065 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7456 ms
ok 117 copying files around # in 10457 ms
ok 118 upconfig # in 2449 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1052 ms
ok 120 downconfig # in 1142 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1344 ms
ok 122 env var ZK_HOST is honored # in 3043 ms
ok 123 Cleanup # in 1794 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 27m 44s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/pcg2xldjb6qxk
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 176 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/176/
No tests ran.
Build Log:
[...truncated 1159 lines...]
# ERROR: Server refused connection at: http://localhost:38823/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:38823/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# POSTing stdin to http://localhost:38823/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# COMMITting Solr index changes to http://localhost:38823/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:38823/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:38823/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 78 verbose echo the Solr response # in 1093 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38823.
#
# ERROR: Server refused connection at: http://localhost:38823/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38823.
#
# ERROR: Server refused connection at: http://localhost:38823/solr/admin/info/system
ok 79 post solr log into solr via cli # in 9069 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14881 ms
ok 82 snapshot lifecycle # in 4586 ms
ok 83 snapshot list # in 3190 ms
ok 84 snapshot describe # in 3203 ms
ok 85 start solr with ssl # in 15581 ms
ok 86 use different hostname when not checking peer-name # in 23809 ms
ok 87 start solr with ssl and auth # in 13829 ms
ok 88 start solr with client truststore and security manager # in 15925 ms
ok 89 start solr with mTLS needed # in 44750 ms
ok 90 start solr with mTLS wanted # in 34063 ms
ok 91 test keystore reload # in 56995 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15426 ms
ok 93 stop command for single port # in 17780 ms
ok 94 check stop command doesn't hang # in 216 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7858 ms
ok 96 deprecated system properties converted to modern properties # in 195326 ms
ok 97 start with custom jetty options # in 7966 ms
ok 98 bootstrapping a configset # in 8848 ms
ok 99 status detects locally running solr # in 9280 ms
ok 100 status with --solr-url from user # in 7795 ms
ok 101 status with --port from user # in 7643 ms
ok 102 multiple connection options are prevented # in 810 ms
ok 103 status with invalid --solr-url from user # in 1176 ms
ok 104 status with --short format # in 196003 ms
ok 105 searching solr via locally executed streaming expression # in 1613 ms
ok 106 searching solr via remotely executed streaming expression # in 1214 ms
ok 107 variable interpolation # in 1583 ms
ok 108 searching solr without credentials fails with error # in 1057 ms
ok 109 --version returns Solr version # in 689 ms
ok 110 version as direct tool call still runs # in 691 ms
ok 111 short help # in 748 ms
ok 112 short help is inferred # in 739 ms
ok 113 long help # in 743 ms
ok 114 running subcommands with zk is prevented # in 676 ms
ok 115 listing out files # in 2105 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7530 ms
ok 117 copying files around # in 10583 ms
ok 118 upconfig # in 2478 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1135 ms
ok 120 downconfig # in 1117 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1386 ms
ok 122 env var ZK_HOST is honored # in 3089 ms
ok 123 Cleanup # in 1817 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 3s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/ibqe3g7r2fxgk
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 175 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/175/
No tests ran.
Build Log:
[...truncated 348 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-04-10 12:31:57.224; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:31:57.227; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:31:57.694; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:31:57.694; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1358 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 44391 is already being used by another process (pid: 881491)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-04-10 12:31:58.897; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:31:58.901; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
not ok 10 post, postlogs and export with basic auth # in 16659 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 44391 is already being used by another process (pid: 881491)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/10-2
# Last output:
# WARN - 2026-04-10 12:32:00.227; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:32:00.231; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 11 package with basic auth # in 12602 ms
ok 12 admin UI static content is excepted from basic auth # in 11243 ms
ok 13 nothing # in 16 ms
ok 14 setting property # in 11749 ms
ok 15 short form of setting property # in 6816 ms
ok 16 connecting to solr via various solr urls and zk hosts # in 9412 ms
ok 17 create for non cloud mode # in 7981 ms
ok 18 create for cloud mode # in 10701 ms
ok 19 multiple connection options are prevented # in 830 ms
ok 20 create collection # in 8094 ms
ok 21 create collection using solrUrl # in 5625 ms
ok 22 create collection using legacy solrUrl # in 5589 ms
ok 23 create collection using Zookeeper # in 5416 ms
ok 24 reject d option with invalid config dir # in 2751 ms
ok 25 accept d option with builtin config # in 5480 ms
ok 26 accept d option with explicit path to config # in 5322 ms
ok 27 accept n option as config name # in 5308 ms
ok 28 allow config reuse when n option specifies same config # in 9521 ms
ok 29 create multisharded collections when s provided # in 5332 ms
ok 30 create replicated collections when rf provided # in 6246 ms
ok 31 can delete collections # in 11219 ms
ok 32 can delete collections with solr-url # in 8586 ms
ok 33 collection delete also deletes zk config # in 7471 ms
ok 34 deletes accompanying zk config with nondefault name # in 7332 ms
ok 35 delete-config option can opt to leave config in zk # in 7110 ms
ok 36 start -e cloud works with --jvm-opts # in 22195 ms
ok 37 start -e cloud works with --no-prompt # in 18911 ms
ok 38 Check export command # in 19831 ms
ok 39 export fails on non cloud mode # in 9648 ms
Tika Server started on port 44396
not ok 40 using curl to extract a single pdf file # in 18766 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 70)
# `assert_output --partial '"numFound":1'' failed
# Waiting up to 180 seconds to see Solr running on port 44391 [|] [/] [-] [\] [|]
# Started Solr server on port 44391 (pid=899953). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:44391/solr/gettingstarted/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c gettingstarted -s http://localhost:44391 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-10 12:36:55.326; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:36:55.331; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'gettingstarted' with 1 shard(s), 1 replica(s) with config-set 'gettingstarted'
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
# "responseHeader":{
# "status":500,
# "QTime":35
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:44396/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:44396/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:6
100 36847 100 15585 100 21262 368k 502k --:--:-- --:--:-- --:--:-- 877k
# 11)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# -- output does not contain substring --
# substring (1 lines):
# "numFound":1
# output (19 lines):
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 218 100 218 0 0 23998 0 --:--:-- --:--:-- --:--:-- 24222
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":6,
# "params":{
# "q":"id:doc1"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
# WARN - 2026-04-10 12:37:03.642; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:37:03.646; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 218 100 218 0 0 23998 0 --:--:-- --:--:-- --:--:-- 24222
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":6,
# "params":{
# "q":"id:doc1"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
not ok 41 using the bin/solr post tool to extract content from pdf # in 19277 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 96)
# `assert_output --partial '"numFound":1'' failed
# Waiting up to 180 seconds to see Solr running on port 44391 [|] [/] [-] [\] [|]
# Started Solr server on port 44391 (pid=900688). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:44391/solr/content_extraction/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c content_extraction -s http://localhost:44391 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-10 12:37:13.791; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:37:13.794; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'content_extraction' with 1 shard(s), 1 replica(s) with config-set 'content_extraction'
#
# -- output does not contain substring --
# substring (1 lines):
# "numFound":1
# output (19 lines):
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 214 100 214 0 0 59116 0 --:--:-- --:--:-- --:--:-- 71333
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":1,
# "params":{
# "q":"*:*"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/41-2
# WARN - 2026-04-10 12:37:23.065; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:37:23.069; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 214 100 214 0 0 59116 0 --:--:-- --:--:-- --:--:-- 71333
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":1,
# "params":{
# "q":"*:*"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
not ok 42 using the bin/solr post tool to crawl web site # in 20452 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 118)
# `assert_output --partial 'POSTed web resource https://solr.apache.org (depth: 0)'' failed
# Waiting up to 180 seconds to see Solr running on port 44391 [|] [/] [-] [\] [|]
# Started Solr server on port 44391 (pid=901473). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:44391/solr/website_extraction/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c website_extraction -s http://localhost:44391 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-10 12:37:33.113; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:37:33.116; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'website_extraction' with 1 shard(s), 1 replica(s) with config-set 'website_extraction'
#
# -- output does not contain substring --
# substring (1 lines):
# POSTed web resource https://solr.apache.org (depth: 0)
# output (196 lines):
# Posting web pages to Solr url http://localhost:44391/solr/website_extraction/update/extract
# Entering auto mode. Indexing pages with content-types corresponding to file endings xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
# Entering crawl at level 0 (1 links total, 1 new)
# PostTool: WARNING: Solr returned an error #500 (Server Error) for url: http://localhost:44391/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: Response: {
# "responseHeader":{
# "status":500,
# "QTime":22
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:44396/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:44396/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# PostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:44391/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: An error occurred while posting https://solr.apache.org
# 0 web pages indexed.
# COMMITting Solr index changes to http://localhost:44391/solr/website_extraction/update/extract...
# Time spent: 0:00:01.167
# --
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/42-3
# WARN - 2026-04-10 12:37:43.604; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-10 12:37:43.607; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# Posting web pages to Solr url http://localhost:44391/solr/website_extraction/update/extract
# Entering auto mode. Indexing pages with content-types corresponding to file endings xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
# Entering crawl at level 0 (1 links total, 1 new)
# PostTool: WARNING: Solr returned an error #500 (Server Error) for url: http://localhost:44391/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: Response: {
# "responseHeader":{
# "status":500,
# "QTime":22
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:44396/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:44396/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# PostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:44391/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: An error occurred while posting https://solr.apache.org
# 0 web pages indexed.
# COMMITting Solr index changes to http://localhost:44391/solr/website_extraction/update/extract...
# Time spent: 0:00:01.167
Stopping Tika Server container
ok 43 healthcheck on cloud solr # in 14445 ms
ok 44 healthcheck errors on standalone solr # in 11487 ms
ok 45 solr help flag prints help # in 757 ms
ok 46 solr with no flags prints help # in 749 ms
ok 47 start help flag prints help # in 150 ms
ok 48 start h flag prints help # in 147 ms
ok 49 stop help flag prints help # in 146 ms
ok 50 restart help flag prints help # in 146 ms
ok 51 status help flag prints help # in 780 ms
ok 52 healthcheck help flag prints help # in 747 ms
ok 53 create help flag prints help # in 764 ms
ok 54 delete help flag prints help # in 769 ms
ok 55 zk help flag prints help # in 764 ms
ok 56 auth help flag prints help # in 766 ms
ok 57 assert help flag prints help # in 743 ms
ok 58 post help flag prints help # in 749 ms
ok 59 SQL Module # in 16109 ms
ok 60 icu collation in analysis-extras module # in 12421 ms
ok 61 Check lifecycle of sentiment classification # in 24310 ms
ok 62 lifecycle of package # in 8237 ms
ok 63 deploying and undeploying a collection level package # in 13478 ms
ok 64 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 65 Affinity placement plugin using sysprop # in 15764 ms
ok 66 Random placement plugin using ENV # in 15540 ms
ok 67 Check help command # in 2476 ms
ok 68 basic post with a type specified # in 5465 ms
ok 69 basic post with collection instead of url parameter # in 4124 ms
ok 70 basic post with solr-url and collection # in 3891 ms
ok 71 basic post WITHOUT a type specified # in 11468 ms
ok 72 crawling a directory as a dry-run # in 793 ms
ok 73 crawling a directory # in 4803 ms
ok 74 crawling a web site # in 34788 ms
ok 75 skipcommit and optimize and delete # in 4908 ms
ok 76 args mode # in 7767 ms
not ok 77 stdin mode # in 6955 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 204)
# `assert_output --partial '"status":0'' failed
#
# -- output does not contain substring --
# substring (1 lines):
# "status":0
# output (35 lines):
# POSTing stdin to http://localhost:44391/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# COMMITting Solr index changes to http://localhost:44391/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:44391/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:44391/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# POSTing stdin to http://localhost:44391/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# COMMITting Solr index changes to http://localhost:44391/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:44391/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:44391/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 78 verbose echo the Solr response # in 1177 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
#
# ERROR: Server refused connection at: http://localhost:44391/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44391.
#
# ERROR: Server refused connection at: http://localhost:44391/solr/admin/info/system
ok 79 post solr log into solr via cli # in 9782 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 15525 ms
ok 82 snapshot lifecycle # in 5025 ms
ok 83 snapshot list # in 3468 ms
ok 84 snapshot describe # in 3381 ms
ok 85 start solr with ssl # in 16527 ms
ok 86 use different hostname when not checking peer-name # in 25678 ms
ok 87 start solr with ssl and auth # in 14590 ms
ok 88 start solr with client truststore and security manager # in 16783 ms
ok 89 start solr with mTLS needed # in 47775 ms
ok 90 start solr with mTLS wanted # in 36923 ms
ok 91 test keystore reload # in 59970 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 16192 ms
ok 93 stop command for single port # in 18418 ms
ok 94 check stop command doesn't hang # in 229 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 8143 ms
ok 96 deprecated system properties converted to modern properties # in 195171 ms
ok 97 start with custom jetty options # in 8168 ms
ok 98 bootstrapping a configset # in 9303 ms
ok 99 status detects locally running solr # in 9650 ms
ok 100 status with --solr-url from user # in 8241 ms
ok 101 status with --port from user # in 8332 ms
ok 102 multiple connection options are prevented # in 881 ms
ok 103 status with invalid --solr-url from user # in 1309 ms
ok 104 status with --short format # in 196092 ms
ok 105 searching solr via locally executed streaming expression # in 1759 ms
ok 106 searching solr via remotely executed streaming expression # in 1259 ms
ok 107 variable interpolation # in 1667 ms
ok 108 searching solr without credentials fails with error # in 1126 ms
ok 109 --version returns Solr version # in 732 ms
ok 110 version as direct tool call still runs # in 731 ms
ok 111 short help # in 823 ms
ok 112 short help is inferred # in 770 ms
ok 113 long help # in 802 ms
ok 114 running subcommands with zk is prevented # in 713 ms
ok 115 listing out files # in 2146 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7998 ms
ok 117 copying files around # in 10991 ms
ok 118 upconfig # in 2559 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1135 ms
ok 120 downconfig # in 1170 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1444 ms
ok 122 env var ZK_HOST is honored # in 3247 ms
ok 123 Cleanup # in 1856 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 11s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/g4uqjhmnyrsiw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 174 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/174/
No tests ran.
Build Log:
[...truncated 348 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-04-09 18:11:37.943; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:11:37.947; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:11:38.378; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:11:38.378; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 8198 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 35987 is already being used by another process (pid: 1526615)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-04-09 18:11:39.443; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:11:39.446; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = ConnectionLoss for /security.json
ok 10 post, postlogs and export with basic auth # in 16262 ms
ok 11 package with basic auth # in 11529 ms
ok 12 admin UI static content is excepted from basic auth # in 9898 ms
ok 13 nothing # in 9 ms
ok 14 setting property # in 9841 ms
ok 15 short form of setting property # in 5903 ms
ok 16 connecting to solr via various solr urls and zk hosts # in 8576 ms
ok 17 create for non cloud mode # in 7423 ms
ok 18 create for cloud mode # in 9257 ms
ok 19 multiple connection options are prevented # in 754 ms
ok 20 create collection # in 6756 ms
ok 21 create collection using solrUrl # in 4854 ms
ok 22 create collection using legacy solrUrl # in 4877 ms
ok 23 create collection using Zookeeper # in 4917 ms
ok 24 reject d option with invalid config dir # in 2481 ms
ok 25 accept d option with builtin config # in 4737 ms
ok 26 accept d option with explicit path to config # in 4896 ms
ok 27 accept n option as config name # in 4820 ms
ok 28 allow config reuse when n option specifies same config # in 8417 ms
ok 29 create multisharded collections when s provided # in 4857 ms
ok 30 create replicated collections when rf provided # in 5668 ms
ok 31 can delete collections # in 9553 ms
ok 32 can delete collections with solr-url # in 7624 ms
ok 33 collection delete also deletes zk config # in 6692 ms
ok 34 deletes accompanying zk config with nondefault name # in 6606 ms
ok 35 delete-config option can opt to leave config in zk # in 6579 ms
ok 36 start -e cloud works with --jvm-opts # in 20552 ms
ok 37 start -e cloud works with --no-prompt # in 17054 ms
ok 38 Check export command # in 18320 ms
ok 39 export fails on non cloud mode # in 8780 ms
Tika Server started on port 35992
not ok 40 using curl to extract a single pdf file # in 17148 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 70)
# `assert_output --partial '"numFound":1'' failed
# Waiting up to 180 seconds to see Solr running on port 35987 [|] [/] [-] [\] [|]
# Started Solr server on port 35987 (pid=1543054). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:35987/solr/gettingstarted/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c gettingstarted -s http://localhost:35987 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-09 18:16:15.034; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:16:15.037; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'gettingstarted' with 1 shard(s), 1 replica(s) with config-set 'gettingstarted'
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
# "responseHeader":{
# "status":500,
# "QTime":31
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:35992/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:35992/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:6
100 36847 100 15585 100 21262 416k 568k --:--:-- --:--:-- --:--:-- 999k
# 11)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# -- output does not contain substring --
# substring (1 lines):
# "numFound":1
# output (19 lines):
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 218 100 218 0 0 29238 0 --:--:-- --:--:-- --:--:-- 31142
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":5,
# "params":{
# "q":"id:doc1"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
# WARN - 2026-04-09 18:16:22.873; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:16:22.877; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 218 100 218 0 0 29238 0 --:--:-- --:--:-- --:--:-- 31142
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":5,
# "params":{
# "q":"id:doc1"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
not ok 41 using the bin/solr post tool to extract content from pdf # in 18078 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 96)
# `assert_output --partial '"numFound":1'' failed
# Waiting up to 180 seconds to see Solr running on port 35987 [|] [/] [-] [\] [|]
# Started Solr server on port 35987 (pid=1543754). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:35987/solr/content_extraction/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c content_extraction -s http://localhost:35987 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-09 18:16:32.247; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:16:32.251; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'content_extraction' with 1 shard(s), 1 replica(s) with config-set 'content_extraction'
#
# -- output does not contain substring --
# substring (1 lines):
# "numFound":1
# output (19 lines):
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 214 100 214 0 0 57900 0 --:--:-- --:--:-- --:--:-- 71333
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":1,
# "params":{
# "q":"*:*"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/41-2
# WARN - 2026-04-09 18:16:41.088; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:16:41.091; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 214 100 214 0 0 57900 0 --:--:-- --:--:-- --:--:-- 71333
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":1,
# "params":{
# "q":"*:*"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
not ok 42 using the bin/solr post tool to crawl web site # in 19006 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 118)
# `assert_output --partial 'POSTed web resource https://solr.apache.org (depth: 0)'' failed
# Waiting up to 180 seconds to see Solr running on port 35987 [|] [/] [-] [\] [|]
# Started Solr server on port 35987 (pid=1544546). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:35987/solr/website_extraction/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c website_extraction -s http://localhost:35987 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-09 18:16:50.495; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:16:50.499; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'website_extraction' with 1 shard(s), 1 replica(s) with config-set 'website_extraction'
#
# -- output does not contain substring --
# substring (1 lines):
# POSTed web resource https://solr.apache.org (depth: 0)
# output (196 lines):
# Posting web pages to Solr url http://localhost:35987/solr/website_extraction/update/extract
# Entering auto mode. Indexing pages with content-types corresponding to file endings xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
# Entering crawl at level 0 (1 links total, 1 new)
# PostTool: WARNING: Solr returned an error #500 (Server Error) for url: http://localhost:35987/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: Response: {
# "responseHeader":{
# "status":500,
# "QTime":21
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:35992/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:35992/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# PostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:35987/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: An error occurred while posting https://solr.apache.org
# 0 web pages indexed.
# COMMITting Solr index changes to http://localhost:35987/solr/website_extraction/update/extract...
# Time spent: 0:00:00.715
# --
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/42-3
# WARN - 2026-04-09 18:17:00.119; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:17:00.122; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# Posting web pages to Solr url http://localhost:35987/solr/website_extraction/update/extract
# Entering auto mode. Indexing pages with content-types corresponding to file endings xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
# Entering crawl at level 0 (1 links total, 1 new)
# PostTool: WARNING: Solr returned an error #500 (Server Error) for url: http://localhost:35987/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: Response: {
# "responseHeader":{
# "status":500,
# "QTime":21
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:35992/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:35992/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# PostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:35987/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: An error occurred while posting https://solr.apache.org
# 0 web pages indexed.
# COMMITting Solr index changes to http://localhost:35987/solr/website_extraction/update/extract...
# Time spent: 0:00:00.715
Stopping Tika Server container
ok 43 healthcheck on cloud solr # in 12451 ms
ok 44 healthcheck errors on standalone solr # in 10326 ms
ok 45 solr help flag prints help # in 682 ms
ok 46 solr with no flags prints help # in 675 ms
ok 47 start help flag prints help # in 133 ms
ok 48 start h flag prints help # in 134 ms
ok 49 stop help flag prints help # in 129 ms
ok 50 restart help flag prints help # in 142 ms
ok 51 status help flag prints help # in 722 ms
ok 52 healthcheck help flag prints help # in 690 ms
ok 53 create help flag prints help # in 686 ms
ok 54 delete help flag prints help # in 690 ms
ok 55 zk help flag prints help # in 690 ms
ok 56 auth help flag prints help # in 698 ms
ok 57 assert help flag prints help # in 691 ms
ok 58 post help flag prints help # in 740 ms
ok 59 SQL Module # in 14538 ms
ok 60 icu collation in analysis-extras module # in 11529 ms
ok 61 Check lifecycle of sentiment classification # in 23759 ms
ok 62 lifecycle of package # in 8039 ms
ok 63 deploying and undeploying a collection level package # in 12791 ms
ok 64 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 65 Affinity placement plugin using sysprop # in 14436 ms
ok 66 Random placement plugin using ENV # in 14397 ms
ok 67 Check help command # in 2094 ms
ok 68 basic post with a type specified # in 4886 ms
ok 69 basic post with collection instead of url parameter # in 3691 ms
ok 70 basic post with solr-url and collection # in 3545 ms
ok 71 basic post WITHOUT a type specified # in 10438 ms
ok 72 crawling a directory as a dry-run # in 723 ms
ok 73 crawling a directory # in 3960 ms
ok 74 crawling a web site # in 34041 ms
ok 75 skipcommit and optimize and delete # in 4017 ms
ok 76 args mode # in 7159 ms
not ok 77 stdin mode # in 5974 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 201)
# `assert_output --partial "Created collection 'test_stdin'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'test_stdin'
# output (13 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:35987/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:35987 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-09 18:20:44.774; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:20:44.777; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:35987/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:35987/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:35987 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-09 18:20:44.774; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-09 18:20:44.777; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:35987/solr/admin/collections
not ok 78 verbose echo the Solr response # in 1078 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
#
# ERROR: Server refused connection at: http://localhost:35987/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:35987.
#
# ERROR: Server refused connection at: http://localhost:35987/solr/admin/info/system
ok 79 post solr log into solr via cli # in 8488 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14197 ms
ok 82 snapshot lifecycle # in 4441 ms
ok 83 snapshot list # in 3063 ms
ok 84 snapshot describe # in 3150 ms
ok 85 start solr with ssl # in 15378 ms
ok 86 use different hostname when not checking peer-name # in 23352 ms
ok 87 start solr with ssl and auth # in 13735 ms
ok 88 start solr with client truststore and security manager # in 15401 ms
ok 89 start solr with mTLS needed # in 43821 ms
ok 90 start solr with mTLS wanted # in 32774 ms
ok 91 test keystore reload # in 55576 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15375 ms
ok 93 stop command for single port # in 17588 ms
ok 94 check stop command doesn't hang # in 200 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7728 ms
ok 96 deprecated system properties converted to modern properties # in 194962 ms
ok 97 start with custom jetty options # in 7715 ms
ok 98 bootstrapping a configset # in 8807 ms
ok 99 status detects locally running solr # in 9042 ms
ok 100 status with --solr-url from user # in 7636 ms
ok 101 status with --port from user # in 7653 ms
ok 102 multiple connection options are prevented # in 808 ms
ok 103 status with invalid --solr-url from user # in 1166 ms
ok 104 status with --short format # in 195787 ms
ok 105 searching solr via locally executed streaming expression # in 1646 ms
ok 106 searching solr via remotely executed streaming expression # in 1179 ms
ok 107 variable interpolation # in 1609 ms
ok 108 searching solr without credentials fails with error # in 1047 ms
ok 109 --version returns Solr version # in 681 ms
ok 110 version as direct tool call still runs # in 679 ms
ok 111 short help # in 719 ms
ok 112 short help is inferred # in 727 ms
ok 113 long help # in 790 ms
ok 114 running subcommands with zk is prevented # in 750 ms
ok 115 listing out files # in 2095 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7423 ms
ok 117 copying files around # in 10495 ms
ok 118 upconfig # in 2471 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1042 ms
ok 120 downconfig # in 1108 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1358 ms
ok 122 env var ZK_HOST is honored # in 3094 ms
ok 123 Cleanup # in 1820 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 27m 34s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/mwsh2647essr6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 173 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/173/
No tests ran.
Build Log:
[...truncated 1167 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:43695/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43695.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43695.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:43695/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:43695 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-08 13:14:30.688; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-08 13:14:30.691; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:43695/solr/admin/collections
not ok 78 verbose echo the Solr response # in 1101 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43695.
#
# ERROR: Server refused connection at: http://localhost:43695/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:43695.
#
# ERROR: Server refused connection at: http://localhost:43695/solr/admin/info/system
ok 79 post solr log into solr via cli # in 9008 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 15052 ms
ok 82 snapshot lifecycle # in 4622 ms
ok 83 snapshot list # in 3227 ms
ok 84 snapshot describe # in 3205 ms
ok 85 start solr with ssl # in 15846 ms
ok 86 use different hostname when not checking peer-name # in 23869 ms
ok 87 start solr with ssl and auth # in 14157 ms
ok 88 start solr with client truststore and security manager # in 16093 ms
ok 89 start solr with mTLS needed # in 46191 ms
ok 90 start solr with mTLS wanted # in 33848 ms
ok 91 test keystore reload # in 56801 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15426 ms
ok 93 stop command for single port # in 17623 ms
ok 94 check stop command doesn't hang # in 230 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7808 ms
ok 96 deprecated system properties converted to modern properties # in 195023 ms
ok 97 start with custom jetty options # in 7789 ms
ok 98 bootstrapping a configset # in 8839 ms
ok 99 status detects locally running solr # in 9156 ms
ok 100 status with --solr-url from user # in 7794 ms
ok 101 status with --port from user # in 7878 ms
ok 102 multiple connection options are prevented # in 812 ms
ok 103 status with invalid --solr-url from user # in 1138 ms
ok 104 status with --short format # in 195893 ms
ok 105 searching solr via locally executed streaming expression # in 1621 ms
ok 106 searching solr via remotely executed streaming expression # in 1215 ms
ok 107 variable interpolation # in 1610 ms
ok 108 searching solr without credentials fails with error # in 1093 ms
ok 109 --version returns Solr version # in 695 ms
ok 110 version as direct tool call still runs # in 716 ms
ok 111 short help # in 785 ms
ok 112 short help is inferred # in 742 ms
ok 113 long help # in 727 ms
ok 114 running subcommands with zk is prevented # in 692 ms
ok 115 listing out files # in 2109 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7480 ms
ok 117 copying files around # in 10586 ms
ok 118 upconfig # in 2545 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1136 ms
ok 120 downconfig # in 1139 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1369 ms
ok 122 env var ZK_HOST is honored # in 3113 ms
ok 123 Cleanup # in 1828 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 18s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vubvbwqb5usvi
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 172 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/172/
No tests ran.
Build Log:
[...truncated 348 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-04-07 16:05:52.729; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:05:52.734; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:05:53.211; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:05:53.211; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 9 create, config, api, and delete with basic auth # in 1396 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 39437 is already being used by another process (pid: 1365594)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1
# Last output:
# WARN - 2026-04-07 16:05:54.481; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:05:54.484; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ok 10 post, postlogs and export with basic auth # in 17828 ms
ok 11 package with basic auth # in 12499 ms
ok 12 admin UI static content is excepted from basic auth # in 10782 ms
ok 13 nothing # in 9 ms
ok 14 setting property # in 11149 ms
ok 15 short form of setting property # in 6835 ms
ok 16 connecting to solr via various solr urls and zk hosts # in 9797 ms
ok 17 create for non cloud mode # in 7996 ms
ok 18 create for cloud mode # in 10171 ms
ok 19 multiple connection options are prevented # in 827 ms
ok 20 create collection # in 7974 ms
ok 21 create collection using solrUrl # in 5517 ms
ok 22 create collection using legacy solrUrl # in 5397 ms
ok 23 create collection using Zookeeper # in 5351 ms
ok 24 reject d option with invalid config dir # in 2940 ms
ok 25 accept d option with builtin config # in 5534 ms
ok 26 accept d option with explicit path to config # in 5411 ms
ok 27 accept n option as config name # in 5393 ms
ok 28 allow config reuse when n option specifies same config # in 9266 ms
ok 29 create multisharded collections when s provided # in 5492 ms
ok 30 create replicated collections when rf provided # in 6267 ms
ok 31 can delete collections # in 10927 ms
ok 32 can delete collections with solr-url # in 8532 ms
ok 33 collection delete also deletes zk config # in 7366 ms
ok 34 deletes accompanying zk config with nondefault name # in 7339 ms
ok 35 delete-config option can opt to leave config in zk # in 7328 ms
ok 36 start -e cloud works with --jvm-opts # in 22503 ms
ok 37 start -e cloud works with --no-prompt # in 44098 ms
ok 38 Check export command # in 20359 ms
ok 39 export fails on non cloud mode # in 9784 ms
Tika Server started on port 39442
not ok 40 using curl to extract a single pdf file # in 18579 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 70)
# `assert_output --partial '"numFound":1'' failed
# Waiting up to 180 seconds to see Solr running on port 39437 [|] [/] [-] [\] [|]
# Started Solr server on port 39437 (pid=1384448). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39437.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39437.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:39437/solr/gettingstarted/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c gettingstarted -s http://localhost:39437 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-07 16:11:16.327; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:11:16.331; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'gettingstarted' with 1 shard(s), 1 replica(s) with config-set 'gettingstarted'
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
# "responseHeader":{
# "status":500,
# "QTime":35
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:39442/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:39442/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:6
100 36847 100 15585 100 21262 370k 505k --:--:-- --:--:-- --:--:-- 877k
# 11)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# -- output does not contain substring --
# substring (1 lines):
# "numFound":1
# output (19 lines):
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 218 100 218 0 0 25677 0 --:--:-- --:--:-- --:--:-- 27250
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":5,
# "params":{
# "q":"id:doc1"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
# --
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/40-1
# WARN - 2026-04-07 16:11:24.500; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:11:24.505; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 218 100 218 0 0 25677 0 --:--:-- --:--:-- --:--:-- 27250
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":5,
# "params":{
# "q":"id:doc1"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
not ok 41 using the bin/solr post tool to extract content from pdf # in 20603 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 96)
# `assert_output --partial '"numFound":1'' failed
# Waiting up to 180 seconds to see Solr running on port 39437 [|] [/] [-] [\] [|]
# Started Solr server on port 39437 (pid=1385175). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39437.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39437.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:39437/solr/content_extraction/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c content_extraction -s http://localhost:39437 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-07 16:11:35.688; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:11:35.692; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'content_extraction' with 1 shard(s), 1 replica(s) with config-set 'content_extraction'
#
# -- output does not contain substring --
# substring (1 lines):
# "numFound":1
# output (19 lines):
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 214 100 214 0 0 51778 0 --:--:-- --:--:-- --:--:-- 53500
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":1,
# "params":{
# "q":"*:*"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
# --
#
# Please find the SOLR_HOME snapshot for failed test #2 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/41-2
# WARN - 2026-04-07 16:11:45.343; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:11:45.347; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# % Total % Received % Xferd Average Speed Time Time Time Current
# Dload Upload Total Spent Left Speed
#
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 214 100 214 0 0 51778 0 --:--:-- --:--:-- --:--:-- 53500
# {
# "responseHeader":{
# "zkConnected":true,
# "status":0,
# "QTime":1,
# "params":{
# "q":"*:*"
# }
# },
# "response":{
# "numFound":0,
# "start":0,
# "numFoundExact":true,
# "docs":[ ]
# }
# }
not ok 42 using the bin/solr post tool to crawl web site # in 22048 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_extraction.bats, line 118)
# `assert_output --partial 'POSTed web resource https://solr.apache.org (depth: 0)'' failed
# Waiting up to 180 seconds to see Solr running on port 39437 [|] [/] [-] [\] [|]
# Started Solr server on port 39437 (pid=1385954). Happy searching!
#
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39437.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39437.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:39437/solr/website_extraction/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c website_extraction -s http://localhost:39437 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-07 16:11:56.051; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:11:56.056; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Created collection 'website_extraction' with 1 shard(s), 1 replica(s) with config-set 'website_extraction'
#
# -- output does not contain substring --
# substring (1 lines):
# POSTed web resource https://solr.apache.org (depth: 0)
# output (196 lines):
# Posting web pages to Solr url http://localhost:39437/solr/website_extraction/update/extract
# Entering auto mode. Indexing pages with content-types corresponding to file endings xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
# Entering crawl at level 0 (1 links total, 1 new)
# PostTool: WARNING: Solr returned an error #500 (Server Error) for url: http://localhost:39437/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: Response: {
# "responseHeader":{
# "status":500,
# "QTime":24
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:39442/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:39442/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# PostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:39437/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: An error occurred while posting https://solr.apache.org
# 0 web pages indexed.
# COMMITting Solr index changes to http://localhost:39437/solr/website_extraction/update/extract...
# Time spent: 0:00:01.617
# --
#
# Please find the SOLR_HOME snapshot for failed test #3 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/42-3
# WARN - 2026-04-07 16:12:07.354; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-07 16:12:07.357; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# Posting web pages to Solr url http://localhost:39437/solr/website_extraction/update/extract
# Entering auto mode. Indexing pages with content-types corresponding to file endings xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
# Entering crawl at level 0 (1 links total, 1 new)
# PostTool: WARNING: Solr returned an error #500 (Server Error) for url: http://localhost:39437/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: Response: {
# "responseHeader":{
# "status":500,
# "QTime":24
# },
# "error":{
# "metadata":{
# "error-class":"org.apache.solr.common.SolrException",
# "root-error-class":"java.net.ConnectException"
# },
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"org.apache.solr.common.SolrException: Error communicating with TikaServer at http://localhost:39442/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:206)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"org.apache.solr.common.SolrException",
# "msg":"Error communicating with TikaServer at http://localhost:39442/tika: ConnectException: Connection refused. Check that a TikaServer is running.",
# "trace":{
# "stackTrace":[
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.callTikaServer(TikaServerExtractionBackend.java:265)",
# "org.apache.solr.handler.extraction.TikaServerExtractionBackend.extractWithSaxHandler(TikaServerExtractionBackend.java:155)",
# "org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:198)",
# "org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:99)",
# "org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:251)",
# "org.apache.solr.core.SolrCore.execute(SolrCore.java:2924)",
# "org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:699)",
# "org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:492)",
# "org.apache.solr.servlet.SolrServlet.dispatch(SolrServlet.java:114)",
# "org.apache.solr.servlet.SolrServlet.service(SolrServlet.java:90)",
# "jakarta.servlet.http.HttpServlet.service(HttpServlet.java:723)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1394)",
# "org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:751)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)",
# "org.apache.solr.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.TracingFilter.doFilter(TracingFilter.java:55)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RateLimitFilter.doFilter(RateLimitFilter.java:50)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.apache.solr.servlet.RequiredSolrRequestFilter.doFilter(RequiredSolrRequestFilter.java:79)",
# "jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)",
# "org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1555)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:823)",
# "org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:440)",
# "org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:470)",
# "org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)",
# "org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)",
# "org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1071)",
# "org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)",
# "org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)",
# "org.eclipse.jetty.server.handler.InetAccessHandler.onConditionsMet(InetAccessHandler.java:50)",
# "org.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)",
# "org.eclipse.jetty.server.Handler$Sequence.handle(Handler.java:805)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler$LastRuleHandler.handle(RewriteHandler.java:159)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.HeaderPatternRule$1.handle(HeaderPatternRule.java:89)",
# "org.eclipse.jetty.rewrite.handler.Rule$Handler.handle(Rule.java:108)",
# "org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:143)",
# "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:611)",
# "org.eclipse.jetty.server.Server.handle(Server.java:182)",
# "org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:677)",
# "org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:416)",
# "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)",
# "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
# "org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)",
# "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)",
# "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ],
# "causedBy":{
# "errorClass":"java.net.ConnectException",
# "msg":"Connection refused",
# "trace":{
# "stackTrace":[
# "java.base/sun.nio.ch.Net.pollConnect(Native Method)",
# "java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)",
# "java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)",
# "org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:373)",
# "org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:346)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:685)",
# "org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:540)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)",
# "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)",
# "org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)",
# "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)",
# "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)",
# "java.base/java.lang.Thread.run(Thread.java:1583)"
# ]
# }
# }
# }
# }
# },
# "code":500
# }
# }
# PostTool: WARNING: IOException while reading response: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:39437/solr/website_extraction/update/extract?literal.id=https%3A%2F%2Fsolr.apache.org&literal.url=https%3A%2F%2Fsolr.apache.org
# PostTool: WARNING: An error occurred while posting https://solr.apache.org
# 0 web pages indexed.
# COMMITting Solr index changes to http://localhost:39437/solr/website_extraction/update/extract...
# Time spent: 0:00:01.617
Stopping Tika Server container
ok 43 healthcheck on cloud solr # in 13869 ms
ok 44 healthcheck errors on standalone solr # in 11567 ms
ok 45 solr help flag prints help # in 776 ms
ok 46 solr with no flags prints help # in 771 ms
ok 47 start help flag prints help # in 148 ms
ok 48 start h flag prints help # in 146 ms
ok 49 stop help flag prints help # in 143 ms
ok 50 restart help flag prints help # in 146 ms
ok 51 status help flag prints help # in 797 ms
ok 52 healthcheck help flag prints help # in 740 ms
ok 53 create help flag prints help # in 737 ms
ok 54 delete help flag prints help # in 771 ms
ok 55 zk help flag prints help # in 763 ms
ok 56 auth help flag prints help # in 768 ms
ok 57 assert help flag prints help # in 748 ms
ok 58 post help flag prints help # in 761 ms
ok 59 SQL Module # in 16032 ms
ok 60 icu collation in analysis-extras module # in 12787 ms
ok 61 Check lifecycle of sentiment classification # in 24471 ms
ok 62 lifecycle of package # in 8509 ms
ok 63 deploying and undeploying a collection level package # in 13927 ms
ok 64 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 65 Affinity placement plugin using sysprop # in 15887 ms
ok 66 Random placement plugin using ENV # in 15921 ms
ok 67 Check help command # in 2496 ms
ok 68 basic post with a type specified # in 5535 ms
ok 69 basic post with collection instead of url parameter # in 4080 ms
ok 70 basic post with solr-url and collection # in 4232 ms
ok 71 basic post WITHOUT a type specified # in 12104 ms
ok 72 crawling a directory as a dry-run # in 826 ms
ok 73 crawling a directory # in 4551 ms
ok 74 crawling a web site # in 33851 ms
ok 75 skipcommit and optimize and delete # in 4670 ms
ok 76 args mode # in 8125 ms
ok 77 stdin mode # in 5516 ms
not ok 78 verbose echo the Solr response # in 11641 ms
# (from function `refute_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 360,
# in test file test/test_post.bats, line 216)
# `refute_output --partial 'ERROR'' failed
#
# -- output should not contain substring --
# substring (1 lines):
# ERROR
# output (47 lines):
# Posting files to [base] url http://localhost:39437/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# <?xml version="1.0" encoding="UTF-8"?>
# <response>
#
# <lst name="responseHeader">
# <int name="rf">1</int>
# <int name="status">0</int>
# <int name="QTime">3156</int>
# </lst>
# </response>
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:39437/solr/monitors_verbose/update...
#
# ERROR: IOException occurred when talking to server at: http://localhost:39437/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:39437/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:496)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.nio.channels.ClosedChannelException
# at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2132)
# at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1170)
# at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:383)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
# at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:210)
# at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
# at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:462)
# at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
# at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
# at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.invokeAsNonBlocking(AdaptiveExecutionStrategy.java:497)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:433)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Posting files to [base] url http://localhost:39437/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# <?xml version="1.0" encoding="UTF-8"?>
# <response>
#
# <lst name="responseHeader">
# <int name="rf">1</int>
# <int name="status">0</int>
# <int name="QTime">3156</int>
# </lst>
# </response>
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:39437/solr/monitors_verbose/update...
#
# ERROR: IOException occurred when talking to server at: http://localhost:39437/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:39437/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:496)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.nio.channels.ClosedChannelException
# at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2132)
# at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1170)
# at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:383)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:514)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
# at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:210)
# at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157)
# at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:462)
# at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
# at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
# at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.invokeAsNonBlocking(AdaptiveExecutionStrategy.java:497)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:433)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
# at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 79 post solr log into solr via cli # in 9787 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 15967 ms
ok 82 snapshot lifecycle # in 5318 ms
ok 83 snapshot list # in 3509 ms
ok 84 snapshot describe # in 3672 ms
ok 85 start solr with ssl # in 17041 ms
ok 86 use different hostname when not checking peer-name # in 25807 ms
ok 87 start solr with ssl and auth # in 15340 ms
ok 88 start solr with client truststore and security manager # in 17118 ms
ok 89 start solr with mTLS needed # in 47000 ms
ok 90 start solr with mTLS wanted # in 37869 ms
ok 91 test keystore reload # in 61146 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 16501 ms
ok 93 stop command for single port # in 19049 ms
ok 94 check stop command doesn't hang # in 231 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 8376 ms
ok 96 deprecated system properties converted to modern properties # in 195400 ms
ok 97 start with custom jetty options # in 8571 ms
ok 98 bootstrapping a configset # in 9568 ms
ok 99 status detects locally running solr # in 9732 ms
ok 100 status with --solr-url from user # in 8457 ms
ok 101 status with --port from user # in 8471 ms
ok 102 multiple connection options are prevented # in 887 ms
ok 103 status with invalid --solr-url from user # in 1243 ms
ok 104 status with --short format # in 196245 ms
ok 105 searching solr via locally executed streaming expression # in 1823 ms
ok 106 searching solr via remotely executed streaming expression # in 1365 ms
ok 107 variable interpolation # in 1790 ms
ok 108 searching solr without credentials fails with error # in 1197 ms
ok 109 --version returns Solr version # in 791 ms
ok 110 version as direct tool call still runs # in 799 ms
ok 111 short help # in 927 ms
ok 112 short help is inferred # in 774 ms
ok 113 long help # in 828 ms
ok 114 running subcommands with zk is prevented # in 792 ms
ok 115 listing out files # in 2188 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 8274 ms
ok 117 copying files around # in 11333 ms
ok 118 upconfig # in 2567 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1217 ms
ok 120 downconfig # in 1275 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1530 ms
ok 122 env var ZK_HOST is honored # in 3579 ms
ok 123 Cleanup # in 1914 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 57s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/fr463y2zrya54
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 171 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/171/
No tests ran.
Build Log:
[...truncated 1167 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:37647/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:37647.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:37647.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:37647/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:37647 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-06 13:29:42.851; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-06 13:29:42.854; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:37647/solr/admin/collections
not ok 78 verbose echo the Solr response # in 1051 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:37647.
#
# ERROR: Server refused connection at: http://localhost:37647/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/78-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:37647.
#
# ERROR: Server refused connection at: http://localhost:37647/solr/admin/info/system
ok 79 post solr log into solr via cli # in 8340 ms
# Docker artifacts saved to:
ok 80 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 81 allowPaths - backup # in 14325 ms
ok 82 snapshot lifecycle # in 4464 ms
ok 83 snapshot list # in 3078 ms
ok 84 snapshot describe # in 3092 ms
ok 85 start solr with ssl # in 14961 ms
ok 86 use different hostname when not checking peer-name # in 23051 ms
ok 87 start solr with ssl and auth # in 13623 ms
ok 88 start solr with client truststore and security manager # in 15365 ms
ok 89 start solr with mTLS needed # in 44413 ms
ok 90 start solr with mTLS wanted # in 32179 ms
ok 91 test keystore reload # in 55087 ms
ok 92 SOLR-11740 check 'solr stop' connection # in 15285 ms
ok 93 stop command for single port # in 17523 ms
ok 94 check stop command doesn't hang # in 201 ms
ok 95 SOLR-16976 solr starts with remote JMX enabled # in 7778 ms
ok 96 deprecated system properties converted to modern properties # in 194965 ms
ok 97 start with custom jetty options # in 7600 ms
ok 98 bootstrapping a configset # in 8860 ms
ok 99 status detects locally running solr # in 9306 ms
ok 100 status with --solr-url from user # in 7758 ms
ok 101 status with --port from user # in 7581 ms
ok 102 multiple connection options are prevented # in 790 ms
ok 103 status with invalid --solr-url from user # in 1205 ms
ok 104 status with --short format # in 195695 ms
ok 105 searching solr via locally executed streaming expression # in 1584 ms
ok 106 searching solr via remotely executed streaming expression # in 1183 ms
ok 107 variable interpolation # in 1560 ms
ok 108 searching solr without credentials fails with error # in 1050 ms
ok 109 --version returns Solr version # in 670 ms
ok 110 version as direct tool call still runs # in 679 ms
ok 111 short help # in 696 ms
ok 112 short help is inferred # in 712 ms
ok 113 long help # in 724 ms
ok 114 running subcommands with zk is prevented # in 680 ms
ok 115 listing out files # in 2060 ms
ok 116 connecting to solr via various solr urls and zk hosts # in 7372 ms
ok 117 copying files around # in 10438 ms
ok 118 upconfig # in 2469 ms
ok 119 SOLR-12429 test upconfig fails with symlink # in 1042 ms
ok 120 downconfig # in 1101 ms
ok 121 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1361 ms
ok 122 env var ZK_HOST is honored # in 3026 ms
ok 123 Cleanup # in 1793 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 27m 48s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/j7zxtxpzgtoi2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 170 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/170/
No tests ran.
Build Log:
[...truncated 1191 lines...]
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:38519/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38519.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:38519.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:38519/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c monitors_verbose -s http://localhost:38519 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-05 15:55:27.245; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-05 15:55:27.248; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:38519/solr/admin/collections
ok 78 post solr log into solr via cli # in 9082 ms
# Docker artifacts saved to:
ok 79 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 80 allowPaths - backup # in 14826 ms
ok 81 snapshot lifecycle # in 4643 ms
ok 82 snapshot list # in 3225 ms
ok 83 snapshot describe # in 3223 ms
ok 84 start solr with ssl # in 15823 ms
ok 85 use different hostname when not checking peer-name # in 24222 ms
ok 86 start solr with ssl and auth # in 14154 ms
ok 87 start solr with client truststore and security manager # in 16246 ms
ok 88 start solr with mTLS needed # in 45142 ms
ok 89 start solr with mTLS wanted # in 33774 ms
ok 90 test keystore reload # in 57879 ms
ok 91 SOLR-11740 check 'solr stop' connection # in 15751 ms
ok 92 stop command for single port # in 18051 ms
ok 93 check stop command doesn't hang # in 218 ms
ok 94 SOLR-16976 solr starts with remote JMX enabled # in 8088 ms
ok 95 deprecated system properties converted to modern properties # in 195004 ms
ok 96 start with custom jetty options # in 8178 ms
ok 97 bootstrapping a configset # in 9124 ms
ok 98 status detects locally running solr # in 9402 ms
ok 99 status with --solr-url from user # in 7936 ms
ok 100 status with --port from user # in 7910 ms
ok 101 multiple connection options are prevented # in 853 ms
ok 102 status with invalid --solr-url from user # in 1215 ms
ok 103 status with --short format # in 195873 ms
ok 104 searching solr via locally executed streaming expression # in 1700 ms
ok 105 searching solr via remotely executed streaming expression # in 1254 ms
ok 106 variable interpolation # in 1665 ms
ok 107 searching solr without credentials fails with error # in 1129 ms
ok 108 --version returns Solr version # in 729 ms
ok 109 version as direct tool call still runs # in 752 ms
ok 110 short help # in 799 ms
ok 111 short help is inferred # in 760 ms
ok 112 long help # in 787 ms
ok 113 running subcommands with zk is prevented # in 700 ms
ok 114 listing out files # in 2105 ms
ok 115 connecting to solr via various solr urls and zk hosts # in 7763 ms
ok 116 copying files around # in 10847 ms
ok 117 upconfig # in 2520 ms
ok 118 SOLR-12429 test upconfig fails with symlink # in 1082 ms
ok 119 downconfig # in 1157 ms
ok 120 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1430 ms
ok 121 env var ZK_HOST is honored # in 3137 ms
ok 122 Cleanup # in 1856 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 336
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 24s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/qu2raeet7erzm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 169 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/169/
No tests ran.
Build Log:
[...truncated 1231 lines...]
BUILD FAILED in 27m 54s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/sj77yrqf6bh2q
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 168 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/168/
No tests ran.
Build Log:
[...truncated 1182 lines...]
# ERROR: Server refused connection at: http://localhost:42949/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:42949/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/76-11
# Last output:
# POSTing stdin to http://localhost:42949/solr/test_stdin/update...
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Connection reset
# COMMITting Solr index changes to http://localhost:42949/solr/test_stdin/update...
#
# ERROR: Server refused connection at: http://localhost:42949/solr/test_stdin/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:42949/solr/test_stdin/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
not ok 77 verbose echo the Solr response # in 1067 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:42949.
#
# ERROR: Server refused connection at: http://localhost:42949/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:42949.
#
# ERROR: Server refused connection at: http://localhost:42949/solr/admin/info/system
ok 78 post solr log into solr via cli # in 8687 ms
# Docker artifacts saved to:
ok 79 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 80 allowPaths - backup # in 14331 ms
ok 81 snapshot lifecycle # in 4481 ms
ok 82 snapshot list # in 3073 ms
ok 83 snapshot describe # in 3095 ms
ok 84 start solr with ssl # in 15100 ms
ok 85 use different hostname when not checking peer-name # in 23234 ms
ok 86 start solr with ssl and auth # in 13694 ms
ok 87 start solr with client truststore and security manager # in 15550 ms
ok 88 start solr with mTLS needed # in 44629 ms
ok 89 start solr with mTLS wanted # in 34082 ms
ok 90 test keystore reload # in 55952 ms
ok 91 SOLR-11740 check 'solr stop' connection # in 15320 ms
ok 92 stop command for single port # in 17556 ms
ok 93 check stop command doesn't hang # in 204 ms
ok 94 SOLR-16976 solr starts with remote JMX enabled # in 7885 ms
ok 95 deprecated system properties converted to modern properties # in 195044 ms
ok 96 start with custom jetty options # in 7724 ms
ok 97 bootstrapping a configset # in 8738 ms
ok 98 status detects locally running solr # in 9145 ms
ok 99 status with --solr-url from user # in 7578 ms
ok 100 status with --port from user # in 7697 ms
ok 101 multiple connection options are prevented # in 795 ms
ok 102 status with invalid --solr-url from user # in 1178 ms
ok 103 status with --short format # in 195845 ms
ok 104 searching solr via locally executed streaming expression # in 1657 ms
ok 105 searching solr via remotely executed streaming expression # in 1179 ms
ok 106 variable interpolation # in 1572 ms
ok 107 searching solr without credentials fails with error # in 1046 ms
ok 108 --version returns Solr version # in 688 ms
ok 109 version as direct tool call still runs # in 681 ms
ok 110 short help # in 713 ms
ok 111 short help is inferred # in 735 ms
ok 112 long help # in 728 ms
ok 113 running subcommands with zk is prevented # in 677 ms
ok 114 listing out files # in 2071 ms
ok 115 connecting to solr via various solr urls and zk hosts # in 7418 ms
ok 116 copying files around # in 10500 ms
ok 117 upconfig # in 2492 ms
ok 118 SOLR-12429 test upconfig fails with symlink # in 1050 ms
ok 119 downconfig # in 1107 ms
ok 120 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1360 ms
ok 121 env var ZK_HOST is honored # in 3056 ms
ok 122 Cleanup # in 1815 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 336
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 11s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/gspmua6ogv626
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 167 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/167/
No tests ran.
Build Log:
[...truncated 1190 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:46347/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/76-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46347.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46347.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:46347/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:46347 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-02 18:14:34.654; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-02 18:14:34.658; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:46347/solr/admin/collections
not ok 77 verbose echo the Solr response # in 1063 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46347.
#
# ERROR: Server refused connection at: http://localhost:46347/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:46347.
#
# ERROR: Server refused connection at: http://localhost:46347/solr/admin/info/system
ok 78 post solr log into solr via cli # in 8753 ms
# Docker artifacts saved to:
ok 79 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 80 allowPaths - backup # in 14666 ms
ok 81 snapshot lifecycle # in 4524 ms
ok 82 snapshot list # in 3170 ms
ok 83 snapshot describe # in 3148 ms
ok 84 start solr with ssl # in 15383 ms
ok 85 use different hostname when not checking peer-name # in 23700 ms
ok 86 start solr with ssl and auth # in 14007 ms
ok 87 start solr with client truststore and security manager # in 15845 ms
ok 88 start solr with mTLS needed # in 44469 ms
ok 89 start solr with mTLS wanted # in 33789 ms
ok 90 test keystore reload # in 56148 ms
ok 91 SOLR-11740 check 'solr stop' connection # in 15439 ms
ok 92 stop command for single port # in 17596 ms
ok 93 check stop command doesn't hang # in 202 ms
ok 94 SOLR-16976 solr starts with remote JMX enabled # in 7752 ms
ok 95 deprecated system properties converted to modern properties # in 195197 ms
ok 96 start with custom jetty options # in 7768 ms
ok 97 bootstrapping a configset # in 8834 ms
ok 98 status detects locally running solr # in 9084 ms
ok 99 status with --solr-url from user # in 7770 ms
ok 100 status with --port from user # in 7703 ms
ok 101 multiple connection options are prevented # in 811 ms
ok 102 status with invalid --solr-url from user # in 1125 ms
ok 103 status with --short format # in 195986 ms
ok 104 searching solr via locally executed streaming expression # in 1613 ms
ok 105 searching solr via remotely executed streaming expression # in 1221 ms
ok 106 variable interpolation # in 1596 ms
ok 107 searching solr without credentials fails with error # in 1060 ms
ok 108 --version returns Solr version # in 692 ms
ok 109 version as direct tool call still runs # in 690 ms
ok 110 short help # in 710 ms
ok 111 short help is inferred # in 727 ms
ok 112 long help # in 743 ms
ok 113 running subcommands with zk is prevented # in 683 ms
ok 114 listing out files # in 2086 ms
ok 115 connecting to solr via various solr urls and zk hosts # in 7487 ms
ok 116 copying files around # in 10518 ms
ok 117 upconfig # in 2481 ms
ok 118 SOLR-12429 test upconfig fails with symlink # in 1063 ms
ok 119 downconfig # in 1119 ms
ok 120 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1352 ms
ok 121 env var ZK_HOST is honored # in 3049 ms
ok 122 Cleanup # in 1800 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 336
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 24s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/3vsgttslmsnea
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 166 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/166/
No tests ran.
Build Log:
[...truncated 421 lines...]
# ERROR: Failed to create collection 'test_args' due to: IOException occurred when talking to server at: http://localhost:34489/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #10 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/75-10
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34489.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34489.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:34489/solr/test_args/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_args -s http://localhost:34489 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-04-01 12:52:58.309; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-04-01 12:52:58.313; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_args' due to: IOException occurred when talking to server at: http://localhost:34489/solr/admin/collections
not ok 76 stdin mode # in 1061 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 201)
# `assert_output --partial "Created collection 'test_stdin'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'test_stdin'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34489.
#
# ERROR: Server refused connection at: http://localhost:34489/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/76-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34489.
#
# ERROR: Server refused connection at: http://localhost:34489/solr/admin/info/system
not ok 77 verbose echo the Solr response # in 1056 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34489.
#
# ERROR: Server refused connection at: http://localhost:34489/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:34489.
#
# ERROR: Server refused connection at: http://localhost:34489/solr/admin/info/system
ok 78 post solr log into solr via cli # in 8479 ms
# Docker artifacts saved to:
ok 79 Docker SolrCloud rolling upgrade # skip Docker image apache/solr-nightly:9.10.0-SNAPSHOT-slim is not available
ok 80 allowPaths - backup # in 14362 ms
ok 81 snapshot lifecycle # in 4440 ms
ok 82 snapshot list # in 3115 ms
ok 83 snapshot describe # in 3105 ms
ok 84 start solr with ssl # in 15058 ms
ok 85 use different hostname when not checking peer-name # in 23452 ms
ok 86 start solr with ssl and auth # in 13675 ms
ok 87 start solr with client truststore and security manager # in 15395 ms
ok 88 start solr with mTLS needed # in 44424 ms
ok 89 start solr with mTLS wanted # in 33326 ms
ok 90 test keystore reload # in 55544 ms
ok 91 SOLR-11740 check 'solr stop' connection # in 15283 ms
ok 92 stop command for single port # in 17516 ms
ok 93 check stop command doesn't hang # in 197 ms
ok 94 SOLR-16976 solr starts with remote JMX enabled # in 7551 ms
ok 95 deprecated system properties converted to modern properties # in 194854 ms
ok 96 start with custom jetty options # in 8118 ms
ok 97 bootstrapping a configset # in 9084 ms
ok 98 status detects locally running solr # in 9070 ms
ok 99 status with --solr-url from user # in 7671 ms
ok 100 status with --port from user # in 7667 ms
ok 101 multiple connection options are prevented # in 792 ms
ok 102 status with invalid --solr-url from user # in 1139 ms
ok 103 status with --short format # in 195687 ms
ok 104 searching solr via locally executed streaming expression # in 1589 ms
ok 105 searching solr via remotely executed streaming expression # in 1171 ms
ok 106 variable interpolation # in 1559 ms
ok 107 searching solr without credentials fails with error # in 1051 ms
ok 108 --version returns Solr version # in 670 ms
ok 109 version as direct tool call still runs # in 678 ms
ok 110 short help # in 699 ms
ok 111 short help is inferred # in 719 ms
ok 112 long help # in 727 ms
ok 113 running subcommands with zk is prevented # in 680 ms
ok 114 listing out files # in 2060 ms
ok 115 connecting to solr via various solr urls and zk hosts # in 7410 ms
ok 116 copying files around # in 10458 ms
ok 117 upconfig # in 2505 ms
ok 118 SOLR-12429 test upconfig fails with symlink # in 1131 ms
ok 119 downconfig # in 1112 ms
ok 120 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1366 ms
ok 121 env var ZK_HOST is honored # in 3051 ms
ok 122 Cleanup # in 1806 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 336
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 1s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/6rpzgud4kjntg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 165 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/165/
No tests ran.
Build Log:
[...truncated 417 lines...]
# ERROR: Server refused connection at: http://localhost:42993/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:42993/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-12
# Last output:
# Posting files to [base] url http://localhost:42993/solr/monitors_verbose/update using content-type application/xml...
# POSTing file monitor.xml to [base]
# PostTool: WARNING: IOException while reading response: java.net.SocketException: Unexpected end of file from server
# 1 files indexed.
# COMMITting Solr index changes to http://localhost:42993/solr/monitors_verbose/update...
#
# ERROR: Server refused connection at: http://localhost:42993/solr/monitors_verbose/update
#
# org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:42993/solr/monitors_verbose/update
# at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:491)
# at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
# at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
# at org.apache.solr.cli.PostTool.commit(PostTool.java:778)
# at org.apache.solr.cli.PostTool.execute(PostTool.java:346)
# at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320)
# at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81)
# at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117)
# Caused by: java.net.ConnectException: Connection refused
# at java.base/sun.nio.ch.Net.pollConnect(Native Method)
# at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
# at java.base/sun.nio.ch.SocketChannelImpl.finishTimedConnect(SocketChannelImpl.java:1191)
# at java.base/sun.nio.ch.SocketChannelImpl.blockingConnect(SocketChannelImpl.java:1233)
# at java.base/sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
# at org.eclipse.jetty.io.ClientConnector.connect(ClientConnector.java:467)
# at org.eclipse.jetty.io.Transport$Socket.connect(Transport.java:175)
# at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:470)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:140)
# at org.eclipse.jetty.http2.client.transport.HttpClientTransportOverHTTP2.connect(HttpClientTransportOverHTTP2.java:128)
# at org.eclipse.jetty.client.HttpClient$1.connect(HttpClient.java:579)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:556)
# at org.eclipse.jetty.client.HttpClient$1.succeeded(HttpClient.java:552)
# at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:186)
# at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:381)
# at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
# at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
# at java.base/java.lang.Thread.run(Thread.java:1583)
ok 78 post solr log into solr via cli # in 8485 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 8 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-03-31T13:15:55.923070906Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-31T13:15:55.923114167Z It should be set to 65000 to avoid operational disruption.
2026-03-31T13:15:55.923119449Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-31T13:15:55.923418356Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-31T13:15:55.923432031Z
2026-03-31T13:15:55.932893978Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-31T13:15:55.940379838Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-31T13:15:55.980351227Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-31T13:15:56.016615264Z WARNING: A command line option has enabled the Security Manager
2026-03-31T13:15:56.016638449Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-31T13:15:57.375929753Z 2026-03-31 13:15:57.372 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-31T13:15:57.641514226Z 2026-03-31 13:15:57.640 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-31T13:15:57.664937291Z 2026-03-31 13:15:57.664 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-31T13:15:57.671451514Z 2026-03-31 13:15:57.670 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-31T13:15:57.671615907Z 2026-03-31 13:15:57.671 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-03-31T13:15:57.671777582Z 2026-03-31 13:15:57.671 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-31T13:15:57.671903881Z 2026-03-31 13:15:57.671 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-31T13:15:57.671230878Z
2026-03-31T13:15:57.674067607Z 2026-03-31 13:15:57.673 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-31T13:15:57.694265208Z 2026-03-31 13:15:57.693 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-31T13:15:57.706729883Z 2026-03-31 13:15:57.706 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-31T13:15:57.706872485Z 2026-03-31 13:15:57.706 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-31T13:15:57.739521473Z 2026-03-31 13:15:57.738 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-31T13:15:58.460187750Z 2026-03-31 13:15:58.459 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-31T13:15:58.689339093Z 2026-03-31 13:15:58.688 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-31T13:15:58.694255611Z 2026-03-31 13:15:58.693 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-03-31T13:15:58.694398977Z 2026-03-31 13:15:58.693 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-03-31T13:15:58.696983791Z 2026-03-31 13:15:58.696 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-03-31T13:15:58.697062407Z 2026-03-31 13:15:58.696 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-03-31T13:15:58.763380220Z 2026-03-31 13:15:58.761 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-03-31T13:15:59.199811774Z 2026-03-31 13:15:59.199 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-03-31T13:15:59.212938315Z 2026-03-31 13:15:59.212 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-31T13:15:59.262521151Z 2026-03-31 13:15:59.261 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-31T13:15:59.290853107Z 2026-03-31 13:15:59.290 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-31T13:15:59.290929653Z 2026-03-31 13:15:59.290 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-31T13:15:59.434247879Z 2026-03-31 13:15:59.433 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-31T13:15:59.455328937Z 2026-03-31 13:15:59.454 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-31T13:15:59.468423837Z 2026-03-31 13:15:59.468 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-03-31T13:15:59.474165690Z 2026-03-31 13:15:59.473 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72155530701242368-solr-node1:8983_solr-n_0000000000) starting
2026-03-31T13:15:59.536021524Z 2026-03-31 13:15:59.535 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-03-31T13:15:59.536644156Z 2026-03-31 13:15:59.536 INFO (OverseerStateUpdate-72155530701242368-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-03-31T13:15:59.547577192Z 2026-03-31 13:15:59.546 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-03-31T13:15:59.586426634Z 2026-03-31 13:15:59.585 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-31T13:16:00.015910310Z 2026-03-31 13:16:00.015 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-31T13:16:00.175676345Z 2026-03-31 13:16:00.175 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-31T13:16:00.425416587Z 2026-03-31 13:16:00.424 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-31T13:16:00.552540814Z 2026-03-31 13:16:00.551 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-31T13:16:00.558660538Z 2026-03-31 13:16:00.558 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_31.request.log
2026-03-31T13:16:00.562490308Z 2026-03-31 13:16:00.562 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-03-31T13:16:00.564213112Z 2026-03-31 13:16:00.563 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4625ms
2026-03-31T13:16:00.747746988Z 2026-03-31 13:16:00.746 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=67
2026-03-31T13:16:00.800332490Z 2026-03-31 13:16:00.799 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=11
2026-03-31T13:16:04.102915473Z 2026-03-31 13:16:04.102 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-03-31T13:16:04.187868894Z 2026-03-31 13:16:04.187 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-31T13:16:07.847893421Z 2026-03-31 13:16:07.846 INFO (qtp985324122-100-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-03-31T13:16:07.875044681Z 2026-03-31 13:16:07.874 INFO (qtp985324122-100-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=7
2026-03-31T13:16:07.901579518Z 2026-03-31 13:16:07.901 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1
2026-03-31T13:16:08.121560461Z 2026-03-31 13:16:08.120 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-03-31T13:16:08.137399430Z 2026-03-31 13:16:08.137 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-03-31T13:16:08.143871932Z 2026-03-31 13:16:08.143 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:08.179849229Z 2026-03-31 13:16:08.177 INFO (OverseerStateUpdate-72155530701242368-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-31T13:16:08.179868444Z "core":"test-collection_shard3_replica_n1",
2026-03-31T13:16:08.179871697Z "node_name":"solr-node1:8983_solr",
2026-03-31T13:16:08.179874617Z "base_url":"http://solr-node1:8983/solr",
2026-03-31T13:16:08.179877260Z "collection":"test-collection",
2026-03-31T13:16:08.179880008Z "shard":"shard3",
2026-03-31T13:16:08.179882536Z "state":"down",
2026-03-31T13:16:08.179887013Z "type":"NRT",
2026-03-31T13:16:08.179889798Z "operation":"ADDREPLICA",
2026-03-31T13:16:08.179892427Z "waitForFinalState":"false"}
2026-03-31T13:16:08.185186050Z 2026-03-31 13:16:08.184 INFO (OverseerStateUpdate-72155530701242368-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-31T13:16:08.185202702Z "core":"test-collection_shard2_replica_n2",
2026-03-31T13:16:08.185206128Z "node_name":"solr-node3:8985_solr",
2026-03-31T13:16:08.185208911Z "base_url":"http://solr-node3:8985/solr",
2026-03-31T13:16:08.185211580Z "collection":"test-collection",
2026-03-31T13:16:08.185214179Z "shard":"shard2",
2026-03-31T13:16:08.185216672Z "state":"down",
2026-03-31T13:16:08.185219423Z "type":"NRT",
2026-03-31T13:16:08.185222071Z "operation":"ADDREPLICA",
2026-03-31T13:16:08.185224662Z "waitForFinalState":"false"}
2026-03-31T13:16:08.189143806Z 2026-03-31 13:16:08.188 INFO (OverseerStateUpdate-72155530701242368-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-31T13:16:08.189156955Z "core":"test-collection_shard1_replica_n4",
2026-03-31T13:16:08.189160220Z "node_name":"solr-node2:8984_solr",
2026-03-31T13:16:08.189173058Z "base_url":"http://solr-node2:8984/solr",
2026-03-31T13:16:08.189175312Z "collection":"test-collection",
2026-03-31T13:16:08.189177544Z "shard":"shard1",
2026-03-31T13:16:08.189179559Z "state":"down",
2026-03-31T13:16:08.189181485Z "type":"NRT",
2026-03-31T13:16:08.189183442Z "operation":"ADDREPLICA",
2026-03-31T13:16:08.189185508Z "waitForFinalState":"false"}
2026-03-31T13:16:08.298635652Z 2026-03-31 13:16:08.298 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:08.359718433Z 2026-03-31 13:16:08.359 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-03-31T13:16:08.404910925Z 2026-03-31 13:16:08.404 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-31T13:16:08.449926611Z 2026-03-31 13:16:08.449 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-03-31T13:16:08.710308014Z 2026-03-31 13:16:08.709 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-31T13:16:08.710721673Z 2026-03-31 13:16:08.709 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-03-31T13:16:08.734165698Z 2026-03-31 13:16:08.733 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-03-31T13:16:09.330568947Z 2026-03-31 13:16:09.329 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-31T13:16:09.377237198Z 2026-03-31 13:16:09.376 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-31T13:16:09.386830230Z 2026-03-31 13:16:09.386 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-31T13:16:09.388795466Z 2026-03-31 13:16:09.387 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-31T13:16:09.388816188Z 2026-03-31 13:16:09.388 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-31T13:16:09.438364507Z 2026-03-31 13:16:09.437 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-31T13:16:09.443009339Z 2026-03-31 13:16:09.442 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-31T13:16:09.445865670Z 2026-03-31 13:16:09.445 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-31T13:16:09.450194226Z 2026-03-31 13:16:09.449 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-31T13:16:09.473879803Z 2026-03-31 13:16:09.473 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-03-31T13:16:09.493413818Z 2026-03-31 13:16:09.493 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-31T13:16:09.500340023Z 2026-03-31 13:16:09.500 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-31T13:16:09.500355162Z 2026-03-31 13:16:09.500 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-31T13:16:09.511637057Z 2026-03-31 13:16:09.510 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-31T13:16:09.513965932Z 2026-03-31 13:16:09.513 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-03-31T13:16:09.515238845Z 2026-03-31 13:16:09.515 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-03-31T13:16:09.528000672Z 2026-03-31 13:16:09.527 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-31T13:16:09.528020398Z 2026-03-31 13:16:09.527 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-31T13:16:09.528037923Z 2026-03-31 13:16:09.527 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-03-31T13:16:09.529010869Z 2026-03-31 13:16:09.528 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-31T13:16:09.529480510Z 2026-03-31 13:16:09.529 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-03-31T13:16:09.532758761Z 2026-03-31 13:16:09.532 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72155530701242368-core_node3-n_0000000000
2026-03-31T13:16:09.537582229Z 2026-03-31 13:16:09.537 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-03-31T13:16:09.641714606Z 2026-03-31 13:16:09.641 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 235] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:09.645380650Z 2026-03-31 13:16:09.645 INFO (qtp985324122-100-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-31T13:16:09.651949846Z 2026-03-31 13:16:09.651 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1293
2026-03-31T13:16:09.757691977Z 2026-03-31 13:16:09.755 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1957960,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-31T13:15:55.703747236Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-03-31T13:16:01.085986535Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-31T13:16:01.086016786Z It should be set to 65000 to avoid operational disruption.
2026-03-31T13:16:01.086019978Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-31T13:16:01.086312817Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-31T13:16:01.086323810Z
2026-03-31T13:16:01.095640343Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-31T13:16:01.103157310Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-31T13:16:01.141047230Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-31T13:16:01.171448429Z WARNING: A command line option has enabled the Security Manager
2026-03-31T13:16:01.171482321Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-31T13:16:02.334338068Z 2026-03-31 13:16:02.330 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-31T13:16:02.603310280Z 2026-03-31 13:16:02.602 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-31T13:16:02.629877510Z 2026-03-31 13:16:02.629 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-31T13:16:02.637412907Z 2026-03-31 13:16:02.636 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-31T13:16:02.637528681Z 2026-03-31 13:16:02.636 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-03-31T13:16:02.637723009Z 2026-03-31 13:16:02.637 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-31T13:16:02.637830921Z 2026-03-31 13:16:02.637 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-31T13:16:02.637165192Z
2026-03-31T13:16:02.640788963Z 2026-03-31 13:16:02.640 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-31T13:16:02.664331090Z 2026-03-31 13:16:02.663 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-31T13:16:02.689381588Z 2026-03-31 13:16:02.688 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-31T13:16:02.841944979Z 2026-03-31 13:16:02.841 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-03-31T13:16:02.862260538Z 2026-03-31 13:16:02.861 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-31T13:16:02.862523784Z 2026-03-31 13:16:02.861 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-31T13:16:02.862657953Z 2026-03-31 13:16:02.861 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-31T13:16:02.973146830Z 2026-03-31 13:16:02.972 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-03-31T13:16:02.973288410Z 2026-03-31 13:16:02.973 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-31T13:16:02.973434841Z 2026-03-31 13:16:02.973 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-31T13:16:03.010416449Z 2026-03-31 13:16:03.009 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-31T13:16:03.802857309Z 2026-03-31 13:16:03.802 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-31T13:16:03.936898585Z 2026-03-31 13:16:03.936 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-31T13:16:03.938684393Z 2026-03-31 13:16:03.936 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-03-31T13:16:03.985207913Z 2026-03-31 13:16:03.982 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-31T13:16:03.993017981Z 2026-03-31 13:16:03.992 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-31T13:16:03.997810600Z 2026-03-31 13:16:03.997 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-31T13:16:03.997897171Z 2026-03-31 13:16:03.997 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-31T13:16:04.029267230Z 2026-03-31 13:16:04.028 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-31T13:16:04.037355313Z 2026-03-31 13:16:04.036 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-03-31T13:16:04.065411269Z 2026-03-31 13:16:04.064 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-31T13:16:04.079416968Z 2026-03-31 13:16:04.078 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-03-31T13:16:04.092715444Z 2026-03-31 13:16:04.092 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-03-31T13:16:04.104050433Z 2026-03-31 13:16:04.103 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-03-31T13:16:04.187834483Z 2026-03-31 13:16:04.187 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-31T13:16:04.256870088Z 2026-03-31 13:16:04.255 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-31T13:16:04.568137103Z 2026-03-31 13:16:04.567 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-31T13:16:04.742123512Z 2026-03-31 13:16:04.741 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-31T13:16:05.003424016Z 2026-03-31 13:16:05.002 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-31T13:16:05.139019814Z 2026-03-31 13:16:05.138 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-31T13:16:05.145349009Z 2026-03-31 13:16:05.145 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_31.request.log
2026-03-31T13:16:05.149211050Z 2026-03-31 13:16:05.148 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-03-31T13:16:05.152953889Z 2026-03-31 13:16:05.152 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4051ms
2026-03-31T13:16:05.309750689Z 2026-03-31 13:16:05.309 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=43
2026-03-31T13:16:05.373545815Z 2026-03-31 13:16:05.373 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-03-31T13:16:08.362121930Z 2026-03-31 13:16:08.361 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-03-31T13:16:08.412092134Z 2026-03-31 13:16:08.411 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-31T13:16:08.466962917Z 2026-03-31 13:16:08.466 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-03-31T13:16:08.730448364Z 2026-03-31 13:16:08.729 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-31T13:16:08.730482207Z 2026-03-31 13:16:08.730 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-03-31T13:16:08.749367022Z 2026-03-31 13:16:08.748 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-03-31T13:16:09.168098623Z 2026-03-31 13:16:09.167 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-31T13:16:09.189224743Z 2026-03-31 13:16:09.188 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-31T13:16:09.192952508Z 2026-03-31 13:16:09.192 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-31T13:16:09.194189103Z 2026-03-31 13:16:09.193 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-31T13:16:09.194266683Z 2026-03-31 13:16:09.193 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-31T13:16:09.270110000Z 2026-03-31 13:16:09.269 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-31T13:16:09.273095303Z 2026-03-31 13:16:09.272 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-31T13:16:09.274913719Z 2026-03-31 13:16:09.274 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-31T13:16:09.276322777Z 2026-03-31 13:16:09.276 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-31T13:16:09.287677610Z 2026-03-31 13:16:09.287 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-03-31T13:16:09.295114511Z 2026-03-31 13:16:09.294 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-31T13:16:09.300882361Z 2026-03-31 13:16:09.300 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-31T13:16:09.301023945Z 2026-03-31 13:16:09.300 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-31T13:16:09.308317006Z 2026-03-31 13:16:09.307 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-31T13:16:09.317038488Z 2026-03-31 13:16:09.316 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-03-31T13:16:09.318330781Z 2026-03-31 13:16:09.318 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-03-31T13:16:09.332210538Z 2026-03-31 13:16:09.331 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-31T13:16:09.332292736Z 2026-03-31 13:16:09.331 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-31T13:16:09.332297799Z 2026-03-31 13:16:09.332 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-03-31T13:16:09.333548393Z 2026-03-31 13:16:09.333 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-31T13:16:09.334790222Z 2026-03-31 13:16:09.334 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-03-31T13:16:09.335824254Z 2026-03-31 13:16:09.335 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72155530701242371-core_node6-n_0000000000
2026-03-31T13:16:09.349038955Z 2026-03-31 13:16:09.348 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-03-31T13:16:09.487912677Z 2026-03-31 13:16:09.487 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:09.491337368Z 2026-03-31 13:16:09.490 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-31T13:16:09.498484179Z 2026-03-31 13:16:09.498 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1137
2026-03-31T13:16:09.641862285Z 2026-03-31 13:16:09.641 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 235] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:09.756542749Z 2026-03-31 13:16:09.756 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1958356,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-31T13:16:00.886609166Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-03-31T13:16:01.314540584Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-31T13:16:01.314576222Z It should be set to 65000 to avoid operational disruption.
2026-03-31T13:16:01.314579436Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-31T13:16:01.314835052Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-31T13:16:01.314838679Z
2026-03-31T13:16:01.323862335Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-31T13:16:01.331152678Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-31T13:16:01.374111203Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-31T13:16:01.404952892Z WARNING: A command line option has enabled the Security Manager
2026-03-31T13:16:01.404978140Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-31T13:16:02.517996831Z 2026-03-31 13:16:02.514 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-31T13:16:02.768763190Z 2026-03-31 13:16:02.768 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-31T13:16:02.795405065Z 2026-03-31 13:16:02.794 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-31T13:16:02.802855608Z 2026-03-31 13:16:02.802 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-31T13:16:02.803016612Z 2026-03-31 13:16:02.802 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-03-31T13:16:02.803201936Z 2026-03-31 13:16:02.802 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-31T13:16:02.803341406Z 2026-03-31 13:16:02.802 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-31T13:16:02.802619555Z
2026-03-31T13:16:02.805821420Z 2026-03-31 13:16:02.805 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-31T13:16:02.827171559Z 2026-03-31 13:16:02.826 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-31T13:16:02.851564778Z 2026-03-31 13:16:02.850 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-31T13:16:02.989970978Z 2026-03-31 13:16:02.989 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-03-31T13:16:03.008062834Z 2026-03-31 13:16:03.007 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-31T13:16:03.008151767Z 2026-03-31 13:16:03.007 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-31T13:16:03.008283755Z 2026-03-31 13:16:03.007 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-31T13:16:03.117908544Z 2026-03-31 13:16:03.117 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-03-31T13:16:03.118004643Z 2026-03-31 13:16:03.117 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-31T13:16:03.118170533Z 2026-03-31 13:16:03.117 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-31T13:16:03.155834282Z 2026-03-31 13:16:03.155 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-31T13:16:03.936737516Z 2026-03-31 13:16:03.936 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-31T13:16:04.077493598Z 2026-03-31 13:16:04.077 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-31T13:16:04.078864789Z 2026-03-31 13:16:04.077 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-03-31T13:16:04.087141278Z 2026-03-31 13:16:04.086 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-31T13:16:04.092152294Z 2026-03-31 13:16:04.091 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-31T13:16:04.096743352Z 2026-03-31 13:16:04.096 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-31T13:16:04.096832184Z 2026-03-31 13:16:04.096 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-31T13:16:04.126324770Z 2026-03-31 13:16:04.125 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-31T13:16:04.136046232Z 2026-03-31 13:16:04.135 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-03-31T13:16:04.155517360Z 2026-03-31 13:16:04.155 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-31T13:16:04.170435058Z 2026-03-31 13:16:04.170 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-03-31T13:16:04.179781134Z 2026-03-31 13:16:04.179 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-03-31T13:16:04.188828572Z 2026-03-31 13:16:04.188 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-31T13:16:04.228129582Z 2026-03-31 13:16:04.227 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-31T13:16:04.503182356Z 2026-03-31 13:16:04.502 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-31T13:16:04.667878233Z 2026-03-31 13:16:04.667 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-31T13:16:04.923285476Z 2026-03-31 13:16:04.922 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-31T13:16:05.053685807Z 2026-03-31 13:16:05.052 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-31T13:16:05.060595834Z 2026-03-31 13:16:05.060 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_31.request.log
2026-03-31T13:16:05.064437303Z 2026-03-31 13:16:05.064 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-03-31T13:16:05.065524854Z 2026-03-31 13:16:05.065 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3735ms
2026-03-31T13:16:06.792966275Z 2026-03-31 13:16:06.787 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=74
2026-03-31T13:16:06.852854027Z 2026-03-31 13:16:06.848 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17
2026-03-31T13:16:08.089842334Z 2026-03-31 13:16:08.089 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=8
2026-03-31T13:16:08.359575281Z 2026-03-31 13:16:08.358 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-03-31T13:16:08.426847578Z 2026-03-31 13:16:08.426 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-31T13:16:08.490195465Z 2026-03-31 13:16:08.489 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config
2026-03-31T13:16:08.784203617Z 2026-03-31 13:16:08.783 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-31T13:16:08.784273952Z 2026-03-31 13:16:08.783 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-03-31T13:16:08.803204178Z 2026-03-31 13:16:08.802 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-03-31T13:16:09.225561107Z 2026-03-31 13:16:09.225 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-31T13:16:09.247030578Z 2026-03-31 13:16:09.246 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-31T13:16:09.254591150Z 2026-03-31 13:16:09.254 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-31T13:16:09.255499178Z 2026-03-31 13:16:09.255 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-31T13:16:09.255622369Z 2026-03-31 13:16:09.255 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-31T13:16:09.294768261Z 2026-03-31 13:16:09.294 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-31T13:16:09.297356543Z 2026-03-31 13:16:09.297 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-31T13:16:09.298972438Z 2026-03-31 13:16:09.298 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-31T13:16:09.302116412Z 2026-03-31 13:16:09.301 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-31T13:16:09.313676664Z 2026-03-31 13:16:09.313 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-03-31T13:16:09.321005846Z 2026-03-31 13:16:09.320 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-31T13:16:09.327440780Z 2026-03-31 13:16:09.327 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-31T13:16:09.327493656Z 2026-03-31 13:16:09.327 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-31T13:16:09.333748806Z 2026-03-31 13:16:09.333 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-31T13:16:09.345847669Z 2026-03-31 13:16:09.345 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-03-31T13:16:09.347330255Z 2026-03-31 13:16:09.347 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-03-31T13:16:09.361366499Z 2026-03-31 13:16:09.360 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-31T13:16:09.361434284Z 2026-03-31 13:16:09.361 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-31T13:16:09.361439679Z 2026-03-31 13:16:09.361 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-03-31T13:16:09.362710807Z 2026-03-31 13:16:09.362 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-31T13:16:09.363560143Z 2026-03-31 13:16:09.363 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-03-31T13:16:09.364577168Z 2026-03-31 13:16:09.364 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72155530701242372-core_node5-n_0000000000
2026-03-31T13:16:09.376014940Z 2026-03-31 13:16:09.375 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-03-31T13:16:09.487908577Z 2026-03-31 13:16:09.487 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:09.491663098Z 2026-03-31 13:16:09.491 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-31T13:16:09.499542101Z 2026-03-31 13:16:09.499 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1142
2026-03-31T13:16:09.641872340Z 2026-03-31 13:16:09.641 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 235] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:09.661796530Z 2026-03-31 13:16:09.661 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-03-31T13:16:09.756642603Z 2026-03-31 13:16:09.756 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-31T13:16:09.760538919Z 2026-03-31 13:16:09.760 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=1655
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1958548,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-31T13:16:01.11067925Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 79 Docker SolrCloud rolling upgrade # in 27796 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 0ed719aef393: Waiting
# 66b76b382631: Waiting
# 69a09c96e8bf: Waiting
# 3aa9120e46f5: Waiting
# 601f2c23751f: Waiting
# 9cdfe5320de8: Waiting
# 4f4fb700ef54: Waiting
# c937aee9a870: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Verifying Checksum
# 69a09c96e8bf: Download complete
# c937aee9a870: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 3aa9120e46f5: Verifying Checksum
# 3aa9120e46f5: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# af6eca94c810: Pull complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 96c832531c38: Pulling fs layer
# ead0ce981274: Pulling fs layer
# f1b445b8fa95: Pulling fs layer
# bfd00f9a8c41: Pulling fs layer
# 6d049ee86af2: Pulling fs layer
# 7e5d2b909f75: Pulling fs layer
# cf6e7c575041: Pulling fs layer
# 0fd922bc13a8: Pulling fs layer
# 787b5e73eccc: Pulling fs layer
# 689ac8ef424a: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# bfd00f9a8c41: Waiting
# 6d049ee86af2: Waiting
# 7e5d2b909f75: Waiting
# cf6e7c575041: Waiting
# 0fd922bc13a8: Waiting
# 689ac8ef424a: Waiting
# 4f4fb700ef54: Waiting
# 787b5e73eccc: Waiting
# ead0ce981274: Verifying Checksum
# ead0ce981274: Download complete
# 96c832531c38: Verifying Checksum
# 96c832531c38: Download complete
# f1b445b8fa95: Verifying Checksum
# f1b445b8fa95: Download complete
# bfd00f9a8c41: Verifying Checksum
# bfd00f9a8c41: Download complete
# cf6e7c575041: Verifying Checksum
# cf6e7c575041: Download complete
# 0fd922bc13a8: Verifying Checksum
# 0fd922bc13a8: Download complete
# 787b5e73eccc: Verifying Checksum
# 787b5e73eccc: Download complete
# 7e5d2b909f75: Verifying Checksum
# 7e5d2b909f75: Download complete
# 689ac8ef424a: Verifying Checksum
# 689ac8ef424a: Download complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 96c832531c38: Pull complete
# ead0ce981274: Pull complete
# f1b445b8fa95: Pull complete
# bfd00f9a8c41: Pull complete
# 6d049ee86af2: Pull complete
# 7e5d2b909f75: Pull complete
# cf6e7c575041: Pull complete
# 0fd922bc13a8: Pull complete
# 787b5e73eccc: Pull complete
# 689ac8ef424a: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:036ba20756ba5bac5aa1d89df251884afdc4553ff218db440b114222628ae9f3
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 9ebe7a20aa3dad4468c37a7963866974b579d8eb4b2761e46db8999309bff7ae
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 1ab8f3d3780759067d23d5afaca747ccbc8d579693573252d8f39ac2b035416c
# Starting solr-node2 connected to first node's ZooKeeper
# ef30ad56aa6a3a588cb326ad720d23c6d5cad91dc1a11d8f6fe0f9c36612557d
# Started solr-node3 connected to first node's ZooKeeper
# 9bb776eddd289d7295fffba8a1872960bbb12f2c26700acfd90545fc11e345f7
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-03-31 13:16:07.929; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-31 13:16:07.962; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 80 allowPaths - backup # in 15074 ms
ok 81 snapshot lifecycle # in 4587 ms
ok 82 snapshot list # in 3169 ms
ok 83 snapshot describe # in 3178 ms
ok 84 start solr with ssl # in 16028 ms
ok 85 use different hostname when not checking peer-name # in 23818 ms
ok 86 start solr with ssl and auth # in 13934 ms
ok 87 start solr with client truststore and security manager # in 15939 ms
ok 88 start solr with mTLS needed # in 44060 ms
ok 89 start solr with mTLS wanted # in 33374 ms
ok 90 test keystore reload # in 55513 ms
ok 91 SOLR-11740 check 'solr stop' connection # in 15294 ms
ok 92 stop command for single port # in 17438 ms
ok 93 check stop command doesn't hang # in 204 ms
ok 94 SOLR-16976 solr starts with remote JMX enabled # in 7739 ms
ok 95 deprecated system properties converted to modern properties # in 195250 ms
ok 96 start with custom jetty options # in 7648 ms
ok 97 bootstrapping a configset # in 8879 ms
ok 98 status detects locally running solr # in 9163 ms
ok 99 status with --solr-url from user # in 7805 ms
ok 100 status with --port from user # in 7779 ms
ok 101 multiple connection options are prevented # in 812 ms
ok 102 status with invalid --solr-url from user # in 1265 ms
ok 103 status with --short format # in 195605 ms
ok 104 searching solr via locally executed streaming expression # in 1601 ms
ok 105 searching solr via remotely executed streaming expression # in 1205 ms
ok 106 variable interpolation # in 1573 ms
ok 107 searching solr without credentials fails with error # in 1062 ms
ok 108 --version returns Solr version # in 702 ms
ok 109 version as direct tool call still runs # in 680 ms
ok 110 short help # in 734 ms
ok 111 short help is inferred # in 724 ms
ok 112 long help # in 726 ms
ok 113 running subcommands with zk is prevented # in 673 ms
ok 114 listing out files # in 2079 ms
ok 115 connecting to solr via various solr urls and zk hosts # in 7375 ms
ok 116 copying files around # in 10515 ms
ok 117 upconfig # in 2435 ms
ok 118 SOLR-12429 test upconfig fails with symlink # in 1138 ms
ok 119 downconfig # in 1103 ms
ok 120 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1359 ms
ok 121 env var ZK_HOST is honored # in 3065 ms
ok 122 Cleanup # in 1795 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 336
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 8s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/75tferelgi7ly
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 164 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/164/
No tests ran.
Build Log:
[...truncated 422 lines...]
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:39883/solr/admin/collections
# --
#
# Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/76-11
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39883.
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39883.
# WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is
# NOT RECOMMENDED for production use.
#
# To turn it off:
# curl http://localhost:39883/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
# Or:
# bin/solr config -c test_stdin -s http://localhost:39883 --action set-user-property --property update.autoCreateFields --value false
# WARN - 2026-03-30 17:50:08.968; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-30 17:50:08.972; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:39883/solr/admin/collections
not ok 77 verbose echo the Solr response # in 1085 ms
# (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247,
# in test file test/test_post.bats, line 210)
# `assert_output --partial "Created collection 'monitors_verbose'"' failed
#
# -- output does not contain substring --
# substring (1 lines):
# Created collection 'monitors_verbose'
# output (3 lines):
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39883.
#
# ERROR: Server refused connection at: http://localhost:39883/solr/admin/info/system
# --
#
# Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/77-12
# Last output:
# Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39883.
#
# ERROR: Server refused connection at: http://localhost:39883/solr/admin/info/system
ok 78 post solr log into solr via cli # in 8649 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-03-30T17:50:32.651793413Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-30T17:50:32.651826283Z It should be set to 65000 to avoid operational disruption.
2026-03-30T17:50:32.651829179Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-30T17:50:32.652079790Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-30T17:50:32.652093842Z
2026-03-30T17:50:32.661096126Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-30T17:50:32.669295970Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-30T17:50:32.712009772Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-30T17:50:32.744801874Z WARNING: A command line option has enabled the Security Manager
2026-03-30T17:50:32.744824828Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-30T17:50:33.962884243Z 2026-03-30 17:50:33.959 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-30T17:50:34.234379724Z 2026-03-30 17:50:34.233 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-30T17:50:34.261768899Z 2026-03-30 17:50:34.261 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-30T17:50:34.268881287Z 2026-03-30 17:50:34.268 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-30T17:50:34.268898816Z 2026-03-30 17:50:34.268 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-03-30T17:50:34.269001508Z 2026-03-30 17:50:34.268 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-30T17:50:34.269146314Z 2026-03-30 17:50:34.268 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-30T17:50:34.268527550Z
2026-03-30T17:50:34.271793040Z 2026-03-30 17:50:34.271 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-30T17:50:34.292790003Z 2026-03-30 17:50:34.292 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-30T17:50:34.305911782Z 2026-03-30 17:50:34.305 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-30T17:50:34.306002333Z 2026-03-30 17:50:34.305 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-30T17:50:34.340065411Z 2026-03-30 17:50:34.339 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-30T17:50:35.057795281Z 2026-03-30 17:50:35.057 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-30T17:50:35.300090585Z 2026-03-30 17:50:35.299 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-30T17:50:35.310604067Z 2026-03-30 17:50:35.310 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-03-30T17:50:35.310709513Z 2026-03-30 17:50:35.310 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-03-30T17:50:35.333636020Z 2026-03-30 17:50:35.333 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-03-30T17:50:35.333727378Z 2026-03-30 17:50:35.333 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-03-30T17:50:35.401354267Z 2026-03-30 17:50:35.400 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-03-30T17:50:35.835502701Z 2026-03-30 17:50:35.835 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-03-30T17:50:35.847466435Z 2026-03-30 17:50:35.847 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-30T17:50:35.885141441Z 2026-03-30 17:50:35.884 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-30T17:50:35.911489012Z 2026-03-30 17:50:35.911 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-30T17:50:35.911541023Z 2026-03-30 17:50:35.911 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-30T17:50:36.065618457Z 2026-03-30 17:50:36.065 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-30T17:50:36.095894340Z 2026-03-30 17:50:36.095 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-30T17:50:36.108239967Z 2026-03-30 17:50:36.107 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-03-30T17:50:36.114253827Z 2026-03-30 17:50:36.113 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72150853307990016-solr-node1:8983_solr-n_0000000000) starting
2026-03-30T17:50:36.211539441Z 2026-03-30 17:50:36.211 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-03-30T17:50:36.212351850Z 2026-03-30 17:50:36.211 INFO (OverseerStateUpdate-72150853307990016-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-03-30T17:50:36.222480766Z 2026-03-30 17:50:36.222 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-03-30T17:50:36.266150010Z 2026-03-30 17:50:36.265 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-30T17:50:36.675944981Z 2026-03-30 17:50:36.675 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-30T17:50:36.847825296Z 2026-03-30 17:50:36.847 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-30T17:50:37.127579658Z 2026-03-30 17:50:37.127 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-30T17:50:37.257883010Z 2026-03-30 17:50:37.257 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-30T17:50:37.263646747Z 2026-03-30 17:50:37.263 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_30.request.log
2026-03-30T17:50:37.267416511Z 2026-03-30 17:50:37.267 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-03-30T17:50:37.268151748Z 2026-03-30 17:50:37.267 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4601ms
2026-03-30T17:50:37.460681856Z 2026-03-30 17:50:37.459 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=42
2026-03-30T17:50:37.588446253Z 2026-03-30 17:50:37.587 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15
2026-03-30T17:50:40.867404546Z 2026-03-30 17:50:40.866 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-03-30T17:50:41.063165792Z 2026-03-30 17:50:41.062 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-30T17:50:45.037566416Z 2026-03-30 17:50:45.036 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8
2026-03-30T17:50:45.062495232Z 2026-03-30 17:50:45.062 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=7
2026-03-30T17:50:45.081932877Z 2026-03-30 17:50:45.081 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1
2026-03-30T17:50:45.321084000Z 2026-03-30 17:50:45.320 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=5
2026-03-30T17:50:45.348301599Z 2026-03-30 17:50:45.347 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-03-30T17:50:45.364655027Z 2026-03-30 17:50:45.364 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-03-30T17:50:45.370395172Z 2026-03-30 17:50:45.370 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:45.416962209Z 2026-03-30 17:50:45.416 INFO (OverseerStateUpdate-72150853307990016-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-30T17:50:45.417033971Z "core":"test-collection_shard3_replica_n1",
2026-03-30T17:50:45.417037216Z "node_name":"solr-node1:8983_solr",
2026-03-30T17:50:45.417039683Z "base_url":"http://solr-node1:8983/solr",
2026-03-30T17:50:45.417041801Z "collection":"test-collection",
2026-03-30T17:50:45.417043878Z "shard":"shard3",
2026-03-30T17:50:45.417045902Z "state":"down",
2026-03-30T17:50:45.417048331Z "type":"NRT",
2026-03-30T17:50:45.417050240Z "operation":"ADDREPLICA",
2026-03-30T17:50:45.417052422Z "waitForFinalState":"false"}
2026-03-30T17:50:45.422362620Z 2026-03-30 17:50:45.422 INFO (OverseerStateUpdate-72150853307990016-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-30T17:50:45.422376077Z "core":"test-collection_shard2_replica_n2",
2026-03-30T17:50:45.422378944Z "node_name":"solr-node3:8985_solr",
2026-03-30T17:50:45.422381126Z "base_url":"http://solr-node3:8985/solr",
2026-03-30T17:50:45.422383238Z "collection":"test-collection",
2026-03-30T17:50:45.422385365Z "shard":"shard2",
2026-03-30T17:50:45.422387773Z "state":"down",
2026-03-30T17:50:45.422389771Z "type":"NRT",
2026-03-30T17:50:45.422392197Z "operation":"ADDREPLICA",
2026-03-30T17:50:45.422394321Z "waitForFinalState":"false"}
2026-03-30T17:50:45.427291325Z 2026-03-30 17:50:45.426 INFO (OverseerStateUpdate-72150853307990016-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-30T17:50:45.427315691Z "core":"test-collection_shard1_replica_n4",
2026-03-30T17:50:45.427318778Z "node_name":"solr-node2:8984_solr",
2026-03-30T17:50:45.427320970Z "base_url":"http://solr-node2:8984/solr",
2026-03-30T17:50:45.427323133Z "collection":"test-collection",
2026-03-30T17:50:45.427326432Z "shard":"shard1",
2026-03-30T17:50:45.427329255Z "state":"down",
2026-03-30T17:50:45.427331932Z "type":"NRT",
2026-03-30T17:50:45.427334770Z "operation":"ADDREPLICA",
2026-03-30T17:50:45.427337883Z "waitForFinalState":"false"}
2026-03-30T17:50:45.538294155Z 2026-03-30 17:50:45.537 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:45.607718843Z 2026-03-30 17:50:45.607 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-03-30T17:50:45.691297003Z 2026-03-30 17:50:45.690 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-30T17:50:45.740346776Z 2026-03-30 17:50:45.739 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-03-30T17:50:46.011741979Z 2026-03-30 17:50:46.011 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-30T17:50:46.011819669Z 2026-03-30 17:50:46.011 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-03-30T17:50:46.031789943Z 2026-03-30 17:50:46.031 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-03-30T17:50:46.634072147Z 2026-03-30 17:50:46.633 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-30T17:50:46.658297253Z 2026-03-30 17:50:46.657 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-30T17:50:46.664918906Z 2026-03-30 17:50:46.664 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-30T17:50:46.665843360Z 2026-03-30 17:50:46.665 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-30T17:50:46.665854997Z 2026-03-30 17:50:46.665 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-30T17:50:46.723979356Z 2026-03-30 17:50:46.723 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-30T17:50:46.726741269Z 2026-03-30 17:50:46.726 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-30T17:50:46.728464842Z 2026-03-30 17:50:46.728 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-30T17:50:46.731838871Z 2026-03-30 17:50:46.731 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-30T17:50:46.745524815Z 2026-03-30 17:50:46.745 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-03-30T17:50:46.754015659Z 2026-03-30 17:50:46.753 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-30T17:50:46.761371368Z 2026-03-30 17:50:46.761 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-30T17:50:46.761427395Z 2026-03-30 17:50:46.761 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-30T17:50:46.768103879Z 2026-03-30 17:50:46.767 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-30T17:50:46.772863428Z 2026-03-30 17:50:46.772 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-03-30T17:50:46.774134247Z 2026-03-30 17:50:46.773 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-03-30T17:50:46.790654962Z 2026-03-30 17:50:46.790 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-30T17:50:46.790678755Z 2026-03-30 17:50:46.790 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-30T17:50:46.790705140Z 2026-03-30 17:50:46.790 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-03-30T17:50:46.791868965Z 2026-03-30 17:50:46.791 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-30T17:50:46.792349637Z 2026-03-30 17:50:46.792 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-03-30T17:50:46.793393891Z 2026-03-30 17:50:46.793 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72150853307990016-core_node3-n_0000000000
2026-03-30T17:50:46.797943967Z 2026-03-30 17:50:46.797 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-03-30T17:50:46.901936358Z 2026-03-30 17:50:46.901 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:46.905581364Z 2026-03-30 17:50:46.905 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-30T17:50:46.912200937Z 2026-03-30 17:50:46.911 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1306
2026-03-30T17:50:46.919025658Z 2026-03-30 17:50:46.918 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-03-30T17:50:47.013154016Z 2026-03-30 17:50:47.012 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:47.015585887Z 2026-03-30 17:50:47.015 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=1682
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 599745,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-30T17:50:32.072122187Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-03-30T17:50:37.957687797Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-30T17:50:37.957720519Z It should be set to 65000 to avoid operational disruption.
2026-03-30T17:50:37.957723829Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-30T17:50:37.957972302Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-30T17:50:37.957975443Z
2026-03-30T17:50:37.972292993Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-30T17:50:37.981078987Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-30T17:50:38.023266642Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-30T17:50:38.055501961Z WARNING: A command line option has enabled the Security Manager
2026-03-30T17:50:38.055520420Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-30T17:50:39.424786830Z 2026-03-30 17:50:39.421 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-30T17:50:39.687683022Z 2026-03-30 17:50:39.687 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-30T17:50:39.713267032Z 2026-03-30 17:50:39.712 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-30T17:50:39.720350467Z 2026-03-30 17:50:39.719 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-30T17:50:39.720496654Z 2026-03-30 17:50:39.719 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-03-30T17:50:39.720633784Z 2026-03-30 17:50:39.719 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-30T17:50:39.720802565Z 2026-03-30 17:50:39.720 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-30T17:50:39.720067452Z
2026-03-30T17:50:39.723229260Z 2026-03-30 17:50:39.722 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-30T17:50:39.745636826Z 2026-03-30 17:50:39.745 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-30T17:50:39.769812117Z 2026-03-30 17:50:39.769 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-30T17:50:39.909694966Z 2026-03-30 17:50:39.909 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-03-30T17:50:39.926605794Z 2026-03-30 17:50:39.926 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-30T17:50:39.926710603Z 2026-03-30 17:50:39.926 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-30T17:50:39.926827375Z 2026-03-30 17:50:39.926 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-30T17:50:40.036149047Z 2026-03-30 17:50:40.035 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-03-30T17:50:40.036274767Z 2026-03-30 17:50:40.036 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-30T17:50:40.036384228Z 2026-03-30 17:50:40.036 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-30T17:50:40.072250474Z 2026-03-30 17:50:40.071 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-30T17:50:40.787026257Z 2026-03-30 17:50:40.786 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-30T17:50:40.935494757Z 2026-03-30 17:50:40.934 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-30T17:50:40.941024798Z 2026-03-30 17:50:40.935 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-03-30T17:50:40.949371282Z 2026-03-30 17:50:40.948 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-30T17:50:40.955179062Z 2026-03-30 17:50:40.954 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-30T17:50:40.959618933Z 2026-03-30 17:50:40.959 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-30T17:50:40.959762111Z 2026-03-30 17:50:40.959 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-30T17:50:40.998387516Z 2026-03-30 17:50:40.997 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-30T17:50:41.007190080Z 2026-03-30 17:50:41.006 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
2026-03-30T17:50:41.030179656Z 2026-03-30 17:50:41.029 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-30T17:50:41.041332525Z 2026-03-30 17:50:41.040 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-03-30T17:50:41.052316177Z 2026-03-30 17:50:41.051 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-03-30T17:50:41.064399819Z 2026-03-30 17:50:41.063 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-30T17:50:41.107930410Z 2026-03-30 17:50:41.107 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-30T17:50:41.439321265Z 2026-03-30 17:50:41.438 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-30T17:50:41.612929716Z 2026-03-30 17:50:41.612 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-30T17:50:41.881955024Z 2026-03-30 17:50:41.881 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-30T17:50:42.014019904Z 2026-03-30 17:50:42.013 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-30T17:50:42.019415404Z 2026-03-30 17:50:42.019 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_30.request.log
2026-03-30T17:50:42.022591723Z 2026-03-30 17:50:42.022 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-03-30T17:50:42.023432754Z 2026-03-30 17:50:42.023 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4044ms
2026-03-30T17:50:42.174310416Z 2026-03-30 17:50:42.173 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=42
2026-03-30T17:50:42.223627178Z 2026-03-30 17:50:42.223 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12
2026-03-30T17:50:45.606916068Z 2026-03-30 17:50:45.606 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-03-30T17:50:45.650346585Z 2026-03-30 17:50:45.649 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-30T17:50:45.700543222Z 2026-03-30 17:50:45.700 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-03-30T17:50:45.984725805Z 2026-03-30 17:50:45.984 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-30T17:50:45.984760561Z 2026-03-30 17:50:45.984 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-03-30T17:50:46.003875054Z 2026-03-30 17:50:46.003 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-03-30T17:50:46.421281063Z 2026-03-30 17:50:46.420 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-30T17:50:46.442220814Z 2026-03-30 17:50:46.441 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-30T17:50:46.447082016Z 2026-03-30 17:50:46.446 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-30T17:50:46.448223311Z 2026-03-30 17:50:46.447 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-30T17:50:46.448236021Z 2026-03-30 17:50:46.447 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-30T17:50:46.510279397Z 2026-03-30 17:50:46.509 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-30T17:50:46.513155239Z 2026-03-30 17:50:46.512 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-30T17:50:46.514902847Z 2026-03-30 17:50:46.514 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-30T17:50:46.516312242Z 2026-03-30 17:50:46.516 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-30T17:50:46.528356714Z 2026-03-30 17:50:46.528 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4}
2026-03-30T17:50:46.535647849Z 2026-03-30 17:50:46.535 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-30T17:50:46.541150166Z 2026-03-30 17:50:46.540 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-30T17:50:46.541181148Z 2026-03-30 17:50:46.540 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-30T17:50:46.549678417Z 2026-03-30 17:50:46.549 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-30T17:50:46.555780590Z 2026-03-30 17:50:46.555 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-03-30T17:50:46.556997700Z 2026-03-30 17:50:46.556 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-03-30T17:50:46.568673932Z 2026-03-30 17:50:46.568 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-30T17:50:46.568727064Z 2026-03-30 17:50:46.568 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-30T17:50:46.568802085Z 2026-03-30 17:50:46.568 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-03-30T17:50:46.569960224Z 2026-03-30 17:50:46.569 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-30T17:50:46.570560347Z 2026-03-30 17:50:46.570 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-03-30T17:50:46.571564441Z 2026-03-30 17:50:46.571 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72150853307990020-core_node6-n_0000000000
2026-03-30T17:50:46.580448023Z 2026-03-30 17:50:46.580 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-03-30T17:50:46.746301868Z 2026-03-30 17:50:46.745 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:46.749875150Z 2026-03-30 17:50:46.749 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-30T17:50:46.758853047Z 2026-03-30 17:50:46.758 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1153
2026-03-30T17:50:46.902559555Z 2026-03-30 17:50:46.902 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:47.013475722Z 2026-03-30 17:50:47.013 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 600157,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-30T17:50:37.684222601Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-03-30T17:50:38.095425908Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-30T17:50:38.095454232Z It should be set to 65000 to avoid operational disruption.
2026-03-30T17:50:38.095457602Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-30T17:50:38.095755835Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-30T17:50:38.095766195Z
2026-03-30T17:50:38.104311466Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-30T17:50:38.111351903Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-30T17:50:38.149840819Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-30T17:50:38.179926172Z WARNING: A command line option has enabled the Security Manager
2026-03-30T17:50:38.179948296Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-30T17:50:39.290205543Z 2026-03-30 17:50:39.286 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-30T17:50:39.537043475Z 2026-03-30 17:50:39.536 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-30T17:50:39.563093969Z 2026-03-30 17:50:39.562 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-30T17:50:39.570331277Z 2026-03-30 17:50:39.569 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-30T17:50:39.570877708Z 2026-03-30 17:50:39.569 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-03-30T17:50:39.570886693Z 2026-03-30 17:50:39.569 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-30T17:50:39.570889693Z 2026-03-30 17:50:39.570 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-30T17:50:39.570071236Z
2026-03-30T17:50:39.573280996Z 2026-03-30 17:50:39.572 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-30T17:50:39.596817191Z 2026-03-30 17:50:39.596 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-30T17:50:39.623400607Z 2026-03-30 17:50:39.622 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-30T17:50:39.763108151Z 2026-03-30 17:50:39.762 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-03-30T17:50:39.780891317Z 2026-03-30 17:50:39.780 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-30T17:50:39.780912705Z 2026-03-30 17:50:39.780 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-30T17:50:39.780915625Z 2026-03-30 17:50:39.780 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-30T17:50:39.891419659Z 2026-03-30 17:50:39.890 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-03-30T17:50:39.891478700Z 2026-03-30 17:50:39.891 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-30T17:50:39.891588870Z 2026-03-30 17:50:39.891 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-30T17:50:39.924462460Z 2026-03-30 17:50:39.923 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-30T17:50:40.619085837Z 2026-03-30 17:50:40.618 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-30T17:50:40.756257273Z 2026-03-30 17:50:40.755 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-30T17:50:40.757618566Z 2026-03-30 17:50:40.755 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-03-30T17:50:40.765965061Z 2026-03-30 17:50:40.765 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-30T17:50:40.770694022Z 2026-03-30 17:50:40.770 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-30T17:50:40.773867873Z 2026-03-30 17:50:40.773 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-30T17:50:40.773928902Z 2026-03-30 17:50:40.773 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-30T17:50:40.803432812Z 2026-03-30 17:50:40.802 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-30T17:50:40.810970368Z 2026-03-30 17:50:40.810 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-03-30T17:50:40.829471856Z 2026-03-30 17:50:40.829 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-30T17:50:40.839397921Z 2026-03-30 17:50:40.838 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-03-30T17:50:40.848148726Z 2026-03-30 17:50:40.847 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-03-30T17:50:40.868226113Z 2026-03-30 17:50:40.867 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-03-30T17:50:40.907523115Z 2026-03-30 17:50:40.906 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-30T17:50:41.063716318Z 2026-03-30 17:50:41.063 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-30T17:50:41.180391868Z 2026-03-30 17:50:41.179 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-30T17:50:41.346082987Z 2026-03-30 17:50:41.345 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-30T17:50:41.602774499Z 2026-03-30 17:50:41.602 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-30T17:50:41.721346516Z 2026-03-30 17:50:41.720 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-30T17:50:41.726909486Z 2026-03-30 17:50:41.726 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_30.request.log
2026-03-30T17:50:41.730323152Z 2026-03-30 17:50:41.729 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-03-30T17:50:41.731203126Z 2026-03-30 17:50:41.730 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3621ms
2026-03-30T17:50:43.815719613Z 2026-03-30 17:50:43.810 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=77
2026-03-30T17:50:43.876025194Z 2026-03-30 17:50:43.875 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18
2026-03-30T17:50:45.607229309Z 2026-03-30 17:50:45.606 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-03-30T17:50:45.669568417Z 2026-03-30 17:50:45.669 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-30T17:50:45.760328225Z 2026-03-30 17:50:45.759 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-03-30T17:50:46.047088392Z 2026-03-30 17:50:46.046 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-30T17:50:46.047125633Z 2026-03-30 17:50:46.046 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-03-30T17:50:46.066284351Z 2026-03-30 17:50:46.065 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-03-30T17:50:46.492175578Z 2026-03-30 17:50:46.491 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-30T17:50:46.513369858Z 2026-03-30 17:50:46.513 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-30T17:50:46.520652897Z 2026-03-30 17:50:46.520 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-30T17:50:46.522048463Z 2026-03-30 17:50:46.521 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-30T17:50:46.522061011Z 2026-03-30 17:50:46.521 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-30T17:50:46.567237156Z 2026-03-30 17:50:46.566 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-30T17:50:46.570055773Z 2026-03-30 17:50:46.569 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-30T17:50:46.572033154Z 2026-03-30 17:50:46.571 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-30T17:50:46.576055312Z 2026-03-30 17:50:46.575 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-30T17:50:46.589588459Z 2026-03-30 17:50:46.588 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-03-30T17:50:46.597604256Z 2026-03-30 17:50:46.597 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-30T17:50:46.603257226Z 2026-03-30 17:50:46.602 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-30T17:50:46.603384296Z 2026-03-30 17:50:46.602 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-30T17:50:46.610195646Z 2026-03-30 17:50:46.609 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-30T17:50:46.618604374Z 2026-03-30 17:50:46.617 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-03-30T17:50:46.620078852Z 2026-03-30 17:50:46.619 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-03-30T17:50:46.631300500Z 2026-03-30 17:50:46.630 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-30T17:50:46.631321188Z 2026-03-30 17:50:46.631 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-30T17:50:46.631333751Z 2026-03-30 17:50:46.631 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-03-30T17:50:46.632478191Z 2026-03-30 17:50:46.632 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-30T17:50:46.632961662Z 2026-03-30 17:50:46.632 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-03-30T17:50:46.634054819Z 2026-03-30 17:50:46.633 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72150853307990019-core_node5-n_0000000000
2026-03-30T17:50:46.641684484Z 2026-03-30 17:50:46.641 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-03-30T17:50:46.746311697Z 2026-03-30 17:50:46.745 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:46.749638455Z 2026-03-30 17:50:46.749 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-30T17:50:46.758723538Z 2026-03-30 17:50:46.758 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1153
2026-03-30T17:50:46.902212904Z 2026-03-30 17:50:46.901 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 236] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-30T17:50:47.013338667Z 2026-03-30 17:50:47.012 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 242] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 600292,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-30T17:50:37.889481487Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 79 Docker SolrCloud rolling upgrade # in 18319 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# Digest: sha256:036ba20756ba5bac5aa1d89df251884afdc4553ff218db440b114222628ae9f3
# Status: Image is up to date for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 7d35516495f4d6464e51bc7de433a2b26322e743a7253d8eeab9e83fa076495d
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# b581055c4f0842b9acf18ddd1991b2642d9be8d2570c3bfe46d7f4364af0e685
# Starting solr-node2 connected to first node's ZooKeeper
# 6b5c1340657ee15740f9d747ed56fb3bc92e542d7a0544a8fd42b0e3826f676a
# Started solr-node3 connected to first node's ZooKeeper
# ba77a60deab5589f7da87ed7ad9ae26dcde2fca49f27388355138187cba1a12d
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-03-30 17:50:45.124; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-30 17:50:45.160; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 80 allowPaths - backup # in 14458 ms
ok 81 snapshot lifecycle # in 4597 ms
ok 82 snapshot list # in 3186 ms
ok 83 snapshot describe # in 3183 ms
ok 84 start solr with ssl # in 15348 ms
ok 85 use different hostname when not checking peer-name # in 23689 ms
ok 86 start solr with ssl and auth # in 14099 ms
ok 87 start solr with client truststore and security manager # in 15934 ms
ok 88 start solr with mTLS needed # in 43812 ms
ok 89 start solr with mTLS wanted # in 34312 ms
ok 90 test keystore reload # in 56789 ms
ok 91 SOLR-11740 check 'solr stop' connection # in 15546 ms
ok 92 stop command for single port # in 17923 ms
ok 93 check stop command doesn't hang # in 205 ms
ok 94 SOLR-16976 solr starts with remote JMX enabled # in 7825 ms
ok 95 deprecated system properties converted to modern properties # in 195211 ms
ok 96 start with custom jetty options # in 7864 ms
ok 97 bootstrapping a configset # in 8899 ms
ok 98 status detects locally running solr # in 9250 ms
ok 99 status with --solr-url from user # in 7834 ms
ok 100 status with --port from user # in 7998 ms
ok 101 multiple connection options are prevented # in 927 ms
ok 102 status with invalid --solr-url from user # in 1309 ms
ok 103 status with --short format # in 195553 ms
ok 104 searching solr via locally executed streaming expression # in 1649 ms
ok 105 searching solr via remotely executed streaming expression # in 1204 ms
ok 106 variable interpolation # in 1580 ms
ok 107 searching solr without credentials fails with error # in 1111 ms
ok 108 --version returns Solr version # in 711 ms
ok 109 version as direct tool call still runs # in 706 ms
ok 110 short help # in 723 ms
ok 111 short help is inferred # in 715 ms
ok 112 long help # in 744 ms
ok 113 running subcommands with zk is prevented # in 709 ms
ok 114 listing out files # in 2082 ms
ok 115 connecting to solr via various solr urls and zk hosts # in 7509 ms
ok 116 copying files around # in 10611 ms
ok 117 upconfig # in 2602 ms
ok 118 SOLR-12429 test upconfig fails with symlink # in 1063 ms
ok 119 downconfig # in 1155 ms
ok 120 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1370 ms
ok 121 env var ZK_HOST is honored # in 3063 ms
ok 122 Cleanup # in 1817 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 336
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 28m 26s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/agdbenpvaa75s
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 163 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/163/
No tests ran.
Build Log:
[...truncated 870 lines...]
BUILD FAILED in 27m 48s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/e3lud75nil7ak
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 162 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/162/
No tests ran.
Build Log:
[...truncated 966 lines...]
BUILD FAILED in 29m
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/s5khumempcg4y
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 161 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/161/
No tests ran.
Build Log:
[...truncated 872 lines...]
BUILD FAILED in 27m 56s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/fyzka6mrfwe3a
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 160 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/160/
No tests ran.
Build Log:
[...truncated 964 lines...]
BUILD FAILED in 28m 38s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/56tcz7np6mwye
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 159 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/159/
No tests ran.
Build Log:
[...truncated 968 lines...]
BUILD FAILED in 30m 13s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/azyd6ygx7gyya
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 158 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/158/
No tests ran.
Build Log:
[...truncated 868 lines...]
BUILD FAILED in 28m 33s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/zgipjqhvaurja
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 157 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/157/
No tests ran.
Build Log:
[...truncated 871 lines...]
BUILD FAILED in 30m 46s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/dsvxpcxvpnwvs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 156 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/156/
No tests ran.
Build Log:
[...truncated 869 lines...]
BUILD FAILED in 28m 23s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/vo3guz4adaucc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 155 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/155/
No tests ran.
Build Log:
[...truncated 921 lines...]
BUILD FAILED in 29m 50s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/gnmgp54oy74di
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 154 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/154/
No tests ran.
Build Log:
[...truncated 948 lines...]
BUILD FAILED in 31m 12s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/sna6kootx3u5m
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 153 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/153/
No tests ran.
Build Log:
[...truncated 898 lines...]
BUILD FAILED in 29m 40s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/erweuhecbgu2a
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 152 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/152/
No tests ran.
Build Log:
[...truncated 347 lines...]
# ERROR: KeeperErrorCode = NoNode for /security.json
# --
#
# Last output:
# WARN - 2026-03-18 12:51:05.764; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-18 12:51:05.767; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-18 12:51:06.215; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-18 12:51:06.215; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: KeeperErrorCode = NoNode for /security.json
not ok 8 create, config, api, and delete with basic auth # in 18119 ms
# (from function `setup' in test file test/test_basic_auth.bats, line 24)
# `solr start -Dsolr.packages.enabled=true' failed
# Starting Solr
#
# Port 41137 is already being used by another process (pid: 2099505)
# Please choose a different port using the -p option.
#
# Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/8-1
# Last output:
# WARN - 2026-03-18 12:51:07.360; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-18 12:51:07.364; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: java.util.concurrent.TimeoutException: Timeout while waiting for Zookeeper Client to connect: 15000 ms
ok 9 post, postlogs and export with basic auth # in 17232 ms
ok 10 package with basic auth # in 11916 ms
ok 11 admin UI static content is excepted from basic auth # in 10472 ms
ok 12 nothing # in 10 ms
ok 13 setting property # in 10562 ms
ok 14 short form of setting property # in 6342 ms
ok 15 connecting to solr via various solr urls and zk hosts # in 9331 ms
ok 16 create for non cloud mode # in 7846 ms
ok 17 create for cloud mode # in 9804 ms
ok 18 multiple connection options are prevented # in 833 ms
ok 19 create collection # in 7530 ms
ok 20 create collection using solrUrl # in 5230 ms
ok 21 create collection using legacy solrUrl # in 5226 ms
ok 22 create collection using Zookeeper # in 5265 ms
ok 23 reject d option with invalid config dir # in 2675 ms
ok 24 accept d option with builtin config # in 5230 ms
ok 25 accept d option with explicit path to config # in 5185 ms
ok 26 accept n option as config name # in 5161 ms
ok 27 allow config reuse when n option specifies same config # in 8854 ms
ok 28 create multisharded collections when s provided # in 5042 ms
ok 29 create replicated collections when rf provided # in 6100 ms
ok 30 can delete collections # in 10195 ms
ok 31 can delete collections with solr-url # in 8023 ms
ok 32 collection delete also deletes zk config # in 7050 ms
ok 33 deletes accompanying zk config with nondefault name # in 7098 ms
ok 34 delete-config option can opt to leave config in zk # in 6932 ms
ok 35 start -e cloud works with --jvm-opts # in 21048 ms
ok 36 start -e cloud works with --no-prompt # in 17943 ms
ok 37 Check export command # in 19036 ms
ok 38 export fails on non cloud mode # in 9131 ms
Tika Server started on port 41142
ok 39 using curl to extract a single pdf file # in 19527 ms
ok 40 using the bin/solr post tool to extract content from pdf # in 20064 ms
ok 41 using the bin/solr post tool to crawl web site # in 21225 ms
Stopping Tika Server container
ok 42 healthcheck on cloud solr # in 13093 ms
ok 43 healthcheck errors on standalone solr # in 10783 ms
ok 44 solr help flag prints help # in 709 ms
ok 45 solr with no flags prints help # in 723 ms
ok 46 start help flag prints help # in 144 ms
ok 47 start h flag prints help # in 151 ms
ok 48 stop help flag prints help # in 141 ms
ok 49 restart help flag prints help # in 146 ms
ok 50 status help flag prints help # in 759 ms
ok 51 healthcheck help flag prints help # in 733 ms
ok 52 create help flag prints help # in 739 ms
ok 53 delete help flag prints help # in 739 ms
ok 54 zk help flag prints help # in 750 ms
ok 55 auth help flag prints help # in 732 ms
ok 56 assert help flag prints help # in 738 ms
ok 57 post help flag prints help # in 735 ms
ok 58 SQL Module # in 15309 ms
ok 59 icu collation in analysis-extras module # in 12028 ms
ok 60 Check lifecycle of sentiment classification # in 23182 ms
ok 61 lifecycle of package # in 8441 ms
ok 62 deploying and undeploying a collection level package # in 13222 ms
ok 63 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github
ok 64 Affinity placement plugin using sysprop # in 15215 ms
ok 65 Random placement plugin using ENV # in 14909 ms
ok 66 Check help command # in 2282 ms
ok 67 basic post with a type specified # in 5154 ms
ok 68 basic post with collection instead of url parameter # in 3866 ms
ok 69 basic post with solr-url and collection # in 3803 ms
ok 70 basic post WITHOUT a type specified # in 11136 ms
ok 71 crawling a directory as a dry-run # in 781 ms
ok 72 crawling a directory # in 4513 ms
ok 73 crawling a web site # in 33850 ms
ok 74 skipcommit and optimize and delete # in 3867 ms
ok 75 args mode # in 6917 ms
ok 76 stdin mode # in 3482 ms
ok 77 verbose echo the Solr response # in 3820 ms
ok 78 post solr log into solr via cli # in 9177 ms
# Test failed - capturing Docker diagnostics
# === docker ps (summary) ===
NAMES STATUS IMAGE PORTS
solr-node3 Up 9 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
solr-node1 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp
# === Docker logs for solr-node1 ===
2026-03-18T13:01:24.649041944Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-18T13:01:24.649075985Z It should be set to 65000 to avoid operational disruption.
2026-03-18T13:01:24.649079197Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-18T13:01:24.649393495Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-18T13:01:24.649404027Z
2026-03-18T13:01:24.659829164Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-18T13:01:24.668307769Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-18T13:01:24.710088952Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-18T13:01:24.743516073Z WARNING: A command line option has enabled the Security Manager
2026-03-18T13:01:24.743541256Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-18T13:01:25.843976990Z 2026-03-18 13:01:25.840 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-18T13:01:26.111603565Z 2026-03-18 13:01:26.110 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-18T13:01:26.136758014Z 2026-03-18 13:01:26.136 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-18T13:01:26.144179861Z 2026-03-18 13:01:26.143 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-18T13:01:26.144272359Z 2026-03-18 13:01:26.143 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983
2026-03-18T13:01:26.144366716Z 2026-03-18 13:01:26.143 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-18T13:01:26.144504255Z 2026-03-18 13:01:26.143 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-18T13:01:26.143946671Z
2026-03-18T13:01:26.147010494Z 2026-03-18 13:01:26.146 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-18T13:01:26.168609020Z 2026-03-18 13:01:26.168 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-18T13:01:26.181646871Z 2026-03-18 13:01:26.181 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-18T13:01:26.181718468Z 2026-03-18 13:01:26.181 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-18T13:01:26.224846870Z 2026-03-18 13:01:26.224 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-18T13:01:26.874829641Z 2026-03-18 13:01:26.874 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-18T13:01:27.113504116Z 2026-03-18 13:01:27.112 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-18T13:01:27.146017780Z 2026-03-18 13:01:27.145 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default
2026-03-18T13:01:27.146123543Z 2026-03-18 13:01:27.145 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg
2026-03-18T13:01:27.149738510Z 2026-03-18 13:01:27.149 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0
2026-03-18T13:01:27.149756558Z 2026-03-18 13:01:27.149 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details.
2026-03-18T13:01:27.191365724Z 2026-03-18 13:01:27.190 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
2026-03-18T13:01:27.662382260Z 2026-03-18 13:01:27.661 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983
2026-03-18T13:01:27.676166887Z 2026-03-18 13:01:27.675 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-18T13:01:27.720260120Z 2026-03-18 13:01:27.719 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-18T13:01:27.747200668Z 2026-03-18 13:01:27.746 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-18T13:01:27.747225636Z 2026-03-18 13:01:27.746 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-18T13:01:27.875558601Z 2026-03-18 13:01:27.875 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-18T13:01:27.906282673Z 2026-03-18 13:01:27.905 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-18T13:01:27.924360421Z 2026-03-18 13:01:27.923 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr
2026-03-18T13:01:27.930672597Z 2026-03-18 13:01:27.930 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72081863546830848-solr-node1:8983_solr-n_0000000000) starting
2026-03-18T13:01:27.997532787Z 2026-03-18 13:01:27.997 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr
2026-03-18T13:01:27.998473869Z 2026-03-18 13:01:27.998 INFO (OverseerStateUpdate-72081863546830848-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr
2026-03-18T13:01:28.010690834Z 2026-03-18 13:01:28.010 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-03-18T13:01:28.055186000Z 2026-03-18 13:01:28.054 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-18T13:01:28.434123841Z 2026-03-18 13:01:28.433 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-18T13:01:28.613061800Z 2026-03-18 13:01:28.612 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-18T13:01:28.886606458Z 2026-03-18 13:01:28.886 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-18T13:01:29.027273253Z 2026-03-18 13:01:29.026 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-18T13:01:29.033277028Z 2026-03-18 13:01:29.032 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_18.request.log
2026-03-18T13:01:29.037438411Z 2026-03-18 13:01:29.036 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983}
2026-03-18T13:01:29.076562475Z 2026-03-18 13:01:29.075 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4372ms
2026-03-18T13:01:29.437903425Z 2026-03-18 13:01:29.436 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=119
2026-03-18T13:01:29.540332699Z 2026-03-18 13:01:29.539 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19
2026-03-18T13:01:32.940865613Z 2026-03-18 13:01:32.940 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-03-18T13:01:32.955687490Z 2026-03-18 13:01:32.955 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-18T13:01:37.032449247Z 2026-03-18 13:01:37.031 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=20
2026-03-18T13:01:37.070159719Z 2026-03-18 13:01:37.069 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=16
2026-03-18T13:01:37.100982952Z 2026-03-18 13:01:37.100 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2
2026-03-18T13:01:37.348439188Z 2026-03-18 13:01:37.348 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=8
2026-03-18T13:01:37.377393196Z 2026-03-18 13:01:37.376 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection
2026-03-18T13:01:37.394245187Z 2026-03-18 13:01:37.393 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children
2026-03-18T13:01:37.401445689Z 2026-03-18 13:01:37.401 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:37.440930701Z 2026-03-18 13:01:37.440 INFO (OverseerStateUpdate-72081863546830848-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-18T13:01:37.440950760Z "core":"test-collection_shard3_replica_n1",
2026-03-18T13:01:37.440954174Z "node_name":"solr-node1:8983_solr",
2026-03-18T13:01:37.440956828Z "base_url":"http://solr-node1:8983/solr",
2026-03-18T13:01:37.440959415Z "collection":"test-collection",
2026-03-18T13:01:37.440962291Z "shard":"shard3",
2026-03-18T13:01:37.440965717Z "state":"down",
2026-03-18T13:01:37.440969316Z "type":"NRT",
2026-03-18T13:01:37.440972888Z "operation":"ADDREPLICA",
2026-03-18T13:01:37.440976688Z "waitForFinalState":"false"}
2026-03-18T13:01:37.446636625Z 2026-03-18 13:01:37.446 INFO (OverseerStateUpdate-72081863546830848-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-18T13:01:37.446653546Z "core":"test-collection_shard2_replica_n2",
2026-03-18T13:01:37.446657090Z "node_name":"solr-node3:8985_solr",
2026-03-18T13:01:37.446660158Z "base_url":"http://solr-node3:8985/solr",
2026-03-18T13:01:37.446662929Z "collection":"test-collection",
2026-03-18T13:01:37.446665519Z "shard":"shard2",
2026-03-18T13:01:37.446668070Z "state":"down",
2026-03-18T13:01:37.446670678Z "type":"NRT",
2026-03-18T13:01:37.446673126Z "operation":"ADDREPLICA",
2026-03-18T13:01:37.446675711Z "waitForFinalState":"false"}
2026-03-18T13:01:37.451317706Z 2026-03-18 13:01:37.450 INFO (OverseerStateUpdate-72081863546830848-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
2026-03-18T13:01:37.451348449Z "core":"test-collection_shard1_replica_n4",
2026-03-18T13:01:37.451351246Z "node_name":"solr-node2:8984_solr",
2026-03-18T13:01:37.451353407Z "base_url":"http://solr-node2:8984/solr",
2026-03-18T13:01:37.451355515Z "collection":"test-collection",
2026-03-18T13:01:37.451357594Z "shard":"shard1",
2026-03-18T13:01:37.451359552Z "state":"down",
2026-03-18T13:01:37.451361567Z "type":"NRT",
2026-03-18T13:01:37.451363521Z "operation":"ADDREPLICA",
2026-03-18T13:01:37.451365584Z "waitForFinalState":"false"}
2026-03-18T13:01:37.559716098Z 2026-03-18 13:01:37.559 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:37.664826708Z 2026-03-18 13:01:37.663 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin
2026-03-18T13:01:37.779516717Z 2026-03-18 13:01:37.778 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-18T13:01:37.844676854Z 2026-03-18 13:01:37.844 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-03-18T13:01:38.237552459Z 2026-03-18 13:01:38.236 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-18T13:01:38.237620210Z 2026-03-18 13:01:38.236 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true
2026-03-18T13:01:38.284971344Z 2026-03-18 13:01:38.284 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/]
2026-03-18T13:01:39.080753577Z 2026-03-18 13:01:39.079 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-18T13:01:39.131256860Z 2026-03-18 13:01:39.130 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-18T13:01:39.140386790Z 2026-03-18 13:01:39.139 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-18T13:01:39.141522318Z 2026-03-18 13:01:39.140 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-18T13:01:39.141641019Z 2026-03-18 13:01:39.141 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-18T13:01:39.210031008Z 2026-03-18 13:01:39.209 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-18T13:01:39.212956077Z 2026-03-18 13:01:39.212 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-18T13:01:39.214805747Z 2026-03-18 13:01:39.214 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-18T13:01:39.216684177Z 2026-03-18 13:01:39.216 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-18T13:01:39.235569528Z 2026-03-18 13:01:39.235 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-03-18T13:01:39.246071985Z 2026-03-18 13:01:39.245 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-18T13:01:39.256372493Z 2026-03-18 13:01:39.255 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-18T13:01:39.256421890Z 2026-03-18 13:01:39.255 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-18T13:01:39.264534470Z 2026-03-18 13:01:39.264 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-18T13:01:39.272202473Z 2026-03-18 13:01:39.271 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm
2026-03-18T13:01:39.274177345Z 2026-03-18 13:01:39.273 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3
2026-03-18T13:01:39.288691337Z 2026-03-18 13:01:39.288 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-18T13:01:39.288818226Z 2026-03-18 13:01:39.288 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-18T13:01:39.288926155Z 2026-03-18 13:01:39.288 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/
2026-03-18T13:01:39.290087233Z 2026-03-18 13:01:39.289 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-18T13:01:39.290862937Z 2026-03-18 13:01:39.290 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas
2026-03-18T13:01:39.292132668Z 2026-03-18 13:01:39.291 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72081863546830848-core_node3-n_0000000000
2026-03-18T13:01:39.298384844Z 2026-03-18 13:01:39.298 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3
2026-03-18T13:01:39.404641894Z 2026-03-18 13:01:39.404 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.407568980Z 2026-03-18 13:01:39.407 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-18T13:01:39.416158265Z 2026-03-18 13:01:39.415 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1754
2026-03-18T13:01:39.425614308Z 2026-03-18 13:01:39.425 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
2026-03-18T13:01:39.522841882Z 2026-03-18 13:01:39.521 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.529858581Z 2026-03-18 13:01:39.529 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2163
# === Docker inspect for solr-node1 ===
{
"Name": "/solr-node1",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2131635,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-18T13:01:24.42366978Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node2 ===
2026-03-18T13:01:29.911937207Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-18T13:01:29.911961169Z It should be set to 65000 to avoid operational disruption.
2026-03-18T13:01:29.911964366Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-18T13:01:29.912304215Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-18T13:01:29.912316530Z
2026-03-18T13:01:29.921803673Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-18T13:01:29.929850234Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-18T13:01:29.975314650Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-18T13:01:30.018596215Z WARNING: A command line option has enabled the Security Manager
2026-03-18T13:01:30.018616832Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-18T13:01:31.180825305Z 2026-03-18 13:01:31.177 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-18T13:01:31.465104906Z 2026-03-18 13:01:31.464 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-18T13:01:31.492089662Z 2026-03-18 13:01:31.491 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-18T13:01:31.499738909Z 2026-03-18 13:01:31.499 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-18T13:01:31.499849764Z 2026-03-18 13:01:31.499 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984
2026-03-18T13:01:31.499946893Z 2026-03-18 13:01:31.499 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-18T13:01:31.500054875Z 2026-03-18 13:01:31.499 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-18T13:01:31.499482816Z
2026-03-18T13:01:31.503393101Z 2026-03-18 13:01:31.502 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-18T13:01:31.527200515Z 2026-03-18 13:01:31.526 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-18T13:01:31.552657810Z 2026-03-18 13:01:31.552 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-18T13:01:31.692396102Z 2026-03-18 13:01:31.691 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-03-18T13:01:31.714586030Z 2026-03-18 13:01:31.714 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-18T13:01:31.714700025Z 2026-03-18 13:01:31.714 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-18T13:01:31.714806142Z 2026-03-18 13:01:31.714 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-18T13:01:31.827070113Z 2026-03-18 13:01:31.826 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-03-18T13:01:31.827199403Z 2026-03-18 13:01:31.826 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-18T13:01:31.827288825Z 2026-03-18 13:01:31.827 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-18T13:01:31.862226414Z 2026-03-18 13:01:31.861 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-18T13:01:32.618519372Z 2026-03-18 13:01:32.617 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-18T13:01:32.815460490Z 2026-03-18 13:01:32.813 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-18T13:01:32.817760820Z 2026-03-18 13:01:32.813 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-03-18T13:01:32.829605952Z 2026-03-18 13:01:32.829 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-18T13:01:32.837338663Z 2026-03-18 13:01:32.836 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-18T13:01:32.844784351Z 2026-03-18 13:01:32.844 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-18T13:01:32.844884252Z 2026-03-18 13:01:32.844 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-18T13:01:32.879936661Z 2026-03-18 13:01:32.879 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-18T13:01:32.890155423Z 2026-03-18 13:01:32.889 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-03-18T13:01:32.916049314Z 2026-03-18 13:01:32.915 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-18T13:01:32.931621672Z 2026-03-18 13:01:32.931 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN
2026-03-18T13:01:32.941741112Z 2026-03-18 13:01:32.941 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-03-18T13:01:32.947246964Z 2026-03-18 13:01:32.946 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr
2026-03-18T13:01:32.956483001Z 2026-03-18 13:01:32.955 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-18T13:01:33.081684811Z 2026-03-18 13:01:33.080 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-18T13:01:33.423361770Z 2026-03-18 13:01:33.422 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-18T13:01:33.648038924Z 2026-03-18 13:01:33.647 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-18T13:01:33.924043228Z 2026-03-18 13:01:33.923 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-18T13:01:34.073196200Z 2026-03-18 13:01:34.072 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-18T13:01:34.078905248Z 2026-03-18 13:01:34.078 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_18.request.log
2026-03-18T13:01:34.083047523Z 2026-03-18 13:01:34.082 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984}
2026-03-18T13:01:34.084540738Z 2026-03-18 13:01:34.084 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4156ms
2026-03-18T13:01:34.244563831Z 2026-03-18 13:01:34.244 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=46
2026-03-18T13:01:34.306052233Z 2026-03-18 13:01:34.305 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=14
2026-03-18T13:01:37.663717769Z 2026-03-18 13:01:37.663 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin
2026-03-18T13:01:37.712619955Z 2026-03-18 13:01:37.711 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-18T13:01:37.770891233Z 2026-03-18 13:01:37.770 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-03-18T13:01:38.060810404Z 2026-03-18 13:01:38.060 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-18T13:01:38.060836055Z 2026-03-18 13:01:38.060 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true
2026-03-18T13:01:38.079101394Z 2026-03-18 13:01:38.078 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/]
2026-03-18T13:01:38.506855658Z 2026-03-18 13:01:38.506 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-18T13:01:38.525952345Z 2026-03-18 13:01:38.525 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-18T13:01:38.529794831Z 2026-03-18 13:01:38.529 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-18T13:01:38.530711164Z 2026-03-18 13:01:38.530 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-18T13:01:38.530762669Z 2026-03-18 13:01:38.530 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-18T13:01:38.572581471Z 2026-03-18 13:01:38.572 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-18T13:01:38.575832238Z 2026-03-18 13:01:38.575 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-18T13:01:38.577734217Z 2026-03-18 13:01:38.577 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-18T13:01:38.579267285Z 2026-03-18 13:01:38.579 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-18T13:01:38.591025404Z 2026-03-18 13:01:38.590 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal}
2026-03-18T13:01:38.598786817Z 2026-03-18 13:01:38.598 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-18T13:01:38.605133574Z 2026-03-18 13:01:38.604 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-18T13:01:38.605155009Z 2026-03-18 13:01:38.604 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-18T13:01:38.612504662Z 2026-03-18 13:01:38.612 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-18T13:01:38.623256750Z 2026-03-18 13:01:38.621 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm
2026-03-18T13:01:38.623990537Z 2026-03-18 13:01:38.623 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1
2026-03-18T13:01:38.636039532Z 2026-03-18 13:01:38.635 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-18T13:01:38.636103984Z 2026-03-18 13:01:38.635 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-18T13:01:38.636108962Z 2026-03-18 13:01:38.635 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/
2026-03-18T13:01:38.637380043Z 2026-03-18 13:01:38.637 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-18T13:01:38.637885605Z 2026-03-18 13:01:38.637 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas
2026-03-18T13:01:38.639028019Z 2026-03-18 13:01:38.638 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72081863546830851-core_node6-n_0000000000
2026-03-18T13:01:38.649671456Z 2026-03-18 13:01:38.649 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1
2026-03-18T13:01:38.757997122Z 2026-03-18 13:01:38.757 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 205] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:38.762081214Z 2026-03-18 13:01:38.761 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-18T13:01:38.772453128Z 2026-03-18 13:01:38.772 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1110
2026-03-18T13:01:38.875255295Z 2026-03-18 13:01:38.874 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 213] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.012241748Z 2026-03-18 13:01:39.011 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.125401394Z 2026-03-18 13:01:39.124 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 226] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.404877704Z 2026-03-18 13:01:39.404 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.522261277Z 2026-03-18 13:01:39.521 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node2 ===
{
"Name": "/solr-node2",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2132033,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-18T13:01:29.661402689Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# === Docker logs for solr-node3 ===
2026-03-18T13:01:30.146003980Z *** [WARN] *** Your open file limit is currently 1024.
2026-03-18T13:01:30.146032196Z It should be set to 65000 to avoid operational disruption.
2026-03-18T13:01:30.146035698Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
2026-03-18T13:01:30.146396811Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag.
2026-03-18T13:01:30.146402950Z
2026-03-18T13:01:30.155384068Z Java 17 detected. Enabled workaround for SOLR-16463
2026-03-18T13:01:30.163698288Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system.
2026-03-18T13:01:30.205299339Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true
2026-03-18T13:01:30.236316042Z WARNING: A command line option has enabled the Security Manager
2026-03-18T13:01:30.236340266Z WARNING: The Security Manager is deprecated and will be removed in a future release
2026-03-18T13:01:31.265173801Z 2026-03-18 13:01:31.261 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8
2026-03-18T13:01:31.539819642Z 2026-03-18 13:01:31.539 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner
2026-03-18T13:01:31.565291960Z 2026-03-18 13:01:31.564 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2026-03-18T13:01:31.572769289Z 2026-03-18 13:01:31.572 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted]
2026-03-18T13:01:31.572889295Z 2026-03-18 13:01:31.572 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985
2026-03-18T13:01:31.573096120Z 2026-03-18 13:01:31.572 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim
2026-03-18T13:01:31.573449848Z 2026-03-18 13:01:31.572 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-03-18T13:01:31.572427547Z
2026-03-18T13:01:31.575530606Z 2026-03-18 13:01:31.575 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log
2026-03-18T13:01:31.598923154Z 2026-03-18 13:01:31.598 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home)
2026-03-18T13:01:31.625326427Z 2026-03-18 13:01:31.624 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-18T13:01:31.774869113Z 2026-03-18 13:01:31.774 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
2026-03-18T13:01:31.795161018Z 2026-03-18 13:01:31.794 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-18T13:01:31.795256931Z 2026-03-18 13:01:31.794 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-18T13:01:31.795414429Z 2026-03-18 13:01:31.794 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
2026-03-18T13:01:31.905886211Z 2026-03-18 13:01:31.905 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
2026-03-18T13:01:31.906144165Z 2026-03-18 13:01:31.905 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default
2026-03-18T13:01:31.906236988Z 2026-03-18 13:01:31.905 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml
2026-03-18T13:01:31.940421465Z 2026-03-18 13:01:31.939 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib]
2026-03-18T13:01:32.684657780Z 2026-03-18 13:01:32.684 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled.
2026-03-18T13:01:32.828069104Z 2026-03-18 13:01:32.827 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure.
2026-03-18T13:01:32.829460884Z 2026-03-18 13:01:32.827 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983
2026-03-18T13:01:32.838762190Z 2026-03-18 13:01:32.838 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-18T13:01:32.843839036Z 2026-03-18 13:01:32.843 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
2026-03-18T13:01:32.846456545Z 2026-03-18 13:01:32.846 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected
2026-03-18T13:01:32.846560528Z 2026-03-18 13:01:32.846 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2026-03-18T13:01:32.877944260Z 2026-03-18 13:01:32.877 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
2026-03-18T13:01:32.886623580Z 2026-03-18 13:01:32.886 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2026-03-18T13:01:32.906821468Z 2026-03-18 13:01:32.906 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
2026-03-18T13:01:32.919554752Z 2026-03-18 13:01:32.919 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN
2026-03-18T13:01:32.932082392Z 2026-03-18 13:01:32.931 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr
2026-03-18T13:01:32.945350672Z 2026-03-18 13:01:32.944 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2026-03-18T13:01:32.956078782Z 2026-03-18 13:01:32.955 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2026-03-18T13:01:32.989407515Z 2026-03-18 13:01:32.988 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info
2026-03-18T13:01:33.288574744Z 2026-03-18 13:01:33.287 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data
2026-03-18T13:01:33.468083931Z 2026-03-18 13:01:33.467 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.
2026-03-18T13:01:33.737218882Z 2026-03-18 13:01:33.736 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-18T13:01:33.865431845Z 2026-03-18 13:01:33.864 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp}
2026-03-18T13:01:33.871372007Z 2026-03-18 13:01:33.870 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_03_18.request.log
2026-03-18T13:01:33.875553665Z 2026-03-18 13:01:33.874 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985}
2026-03-18T13:01:33.876374908Z 2026-03-18 13:01:33.876 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3713ms
2026-03-18T13:01:35.870215890Z 2026-03-18 13:01:35.863 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=126
2026-03-18T13:01:35.940280991Z 2026-03-18 13:01:35.939 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=27
2026-03-18T13:01:37.666633584Z 2026-03-18 13:01:37.665 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin
2026-03-18T13:01:37.748573426Z 2026-03-18 13:01:37.748 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0
2026-03-18T13:01:37.833424007Z 2026-03-18 13:01:37.832 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config
2026-03-18T13:01:38.187687284Z 2026-03-18 13:01:38.187 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id
2026-03-18T13:01:38.187720341Z 2026-03-18 13:01:38.187 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true
2026-03-18T13:01:38.210538410Z 2026-03-18 13:01:38.209 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/]
2026-03-18T13:01:38.703603375Z 2026-03-18 13:01:38.703 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
2026-03-18T13:01:38.725965817Z 2026-03-18 13:01:38.725 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
2026-03-18T13:01:38.733044398Z 2026-03-18 13:01:38.732 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
2026-03-18T13:01:38.734339847Z 2026-03-18 13:01:38.733 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
2026-03-18T13:01:38.734361388Z 2026-03-18 13:01:38.733 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms;
2026-03-18T13:01:38.780405605Z 2026-03-18 13:01:38.779 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts
2026-03-18T13:01:38.783500271Z 2026-03-18 13:01:38.783 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts
2026-03-18T13:01:38.785619705Z 2026-03-18 13:01:38.785 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml
2026-03-18T13:01:38.787292816Z 2026-03-18 13:01:38.787 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
2026-03-18T13:01:38.803864786Z 2026-03-18 13:01:38.803 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
2026-03-18T13:01:38.812537593Z 2026-03-18 13:01:38.811 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
2026-03-18T13:01:38.848493548Z 2026-03-18 13:01:38.848 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done.
2026-03-18T13:01:38.848516839Z 2026-03-18 13:01:38.848 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2026-03-18T13:01:38.855505159Z 2026-03-18 13:01:38.855 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
2026-03-18T13:01:38.875805108Z 2026-03-18 13:01:38.875 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm
2026-03-18T13:01:38.877317446Z 2026-03-18 13:01:38.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2
2026-03-18T13:01:38.891587074Z 2026-03-18 13:01:38.891 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
2026-03-18T13:01:38.891608456Z 2026-03-18 13:01:38.891 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
2026-03-18T13:01:38.891612212Z 2026-03-18 13:01:38.891 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/
2026-03-18T13:01:38.893148086Z 2026-03-18 13:01:38.892 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
2026-03-18T13:01:38.893641438Z 2026-03-18 13:01:38.893 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas
2026-03-18T13:01:38.894704148Z 2026-03-18 13:01:38.894 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72081863546830852-core_node5-n_0000000000
2026-03-18T13:01:38.904387092Z 2026-03-18 13:01:38.903 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2
2026-03-18T13:01:39.012348581Z 2026-03-18 13:01:39.011 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.015630581Z 2026-03-18 13:01:39.015 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary
2026-03-18T13:01:39.023221835Z 2026-03-18 13:01:39.022 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1359
2026-03-18T13:01:39.125592982Z 2026-03-18 13:01:39.125 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 226] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.404859617Z 2026-03-18 13:01:39.404 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 241] for collection [test-collection] has occurred - updating... (live nodes size: [3])
2026-03-18T13:01:39.522224241Z 2026-03-18 13:01:39.521 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 247] for collection [test-collection] has occurred - updating... (live nodes size: [3])
# === Docker inspect for solr-node3 ===
{
"Name": "/solr-node3",
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2132216,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-03-18T13:01:29.908892232Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Ports": {
"8983/tcp": null
}
}
# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 79 Docker SolrCloud rolling upgrade # in 29205 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
# `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# 601f2c23751f: Waiting
# 0ed719aef393: Waiting
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 9cdfe5320de8: Waiting
# 66b76b382631: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 3aa9120e46f5: Download complete
# 9cdfe5320de8: Verifying Checksum
# 9cdfe5320de8: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# af6eca94c810: Pull complete
# 4f4fb700ef54: Download complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim
# 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# 96c832531c38: Pulling fs layer
# ead0ce981274: Pulling fs layer
# f1b445b8fa95: Pulling fs layer
# bfd00f9a8c41: Pulling fs layer
# 6d049ee86af2: Pulling fs layer
# 949290dfdb00: Pulling fs layer
# db80cb1d975b: Pulling fs layer
# 53c536199aed: Pulling fs layer
# a3d67f63bbd2: Pulling fs layer
# aafab940886d: Pulling fs layer
# 4f4fb700ef54: Pulling fs layer
# db80cb1d975b: Waiting
# aafab940886d: Waiting
# 4f4fb700ef54: Waiting
# 53c536199aed: Waiting
# a3d67f63bbd2: Waiting
# 6d049ee86af2: Waiting
# bfd00f9a8c41: Waiting
# 949290dfdb00: Waiting
# 96c832531c38: Download complete
# f1b445b8fa95: Verifying Checksum
# f1b445b8fa95: Download complete
# bfd00f9a8c41: Verifying Checksum
# bfd00f9a8c41: Download complete
# 6d049ee86af2: Verifying Checksum
# 6d049ee86af2: Download complete
# ead0ce981274: Verifying Checksum
# ead0ce981274: Download complete
# 53c536199aed: Download complete
# db80cb1d975b: Verifying Checksum
# db80cb1d975b: Download complete
# 96c832531c38: Pull complete
# a3d67f63bbd2: Verifying Checksum
# a3d67f63bbd2: Download complete
# aafab940886d: Verifying Checksum
# aafab940886d: Download complete
# 949290dfdb00: Verifying Checksum
# 949290dfdb00: Download complete
# 4f4fb700ef54: Verifying Checksum
# ead0ce981274: Pull complete
# f1b445b8fa95: Pull complete
# bfd00f9a8c41: Pull complete
# 6d049ee86af2: Pull complete
# 949290dfdb00: Pull complete
# db80cb1d975b: Pull complete
# 53c536199aed: Pull complete
# a3d67f63bbd2: Pull complete
# aafab940886d: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:f040d7d59f20137b0fbe34ed60d8fe78e710b7249b612b0616865dbf6762b4a4
# Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim
# docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim
# 70ebadf480a87e3a3262d4021119326554fdd5878608bf371bc55fee4cf47759
# solr-data1
# solr-data2
# solr-data3
# Starting solr-node1 with embedded ZooKeeper
# 7e9f5f4f26846822567474834437373475840de8b3d7d6b6e5bbae872af6facb
# Starting solr-node2 connected to first node's ZooKeeper
# f730e81c902c4d7224b23caa048b0bf66a4521824fa4f5a841c37c24295ff245
# Started solr-node3 connected to first node's ZooKeeper
# 1cba3a024ef4b98fdfc5cffe166da255c67eb92dbdcb87528c17433a2bcc0820
# Creating a Collection
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983.
# WARN - 2026-03-18 13:01:37.132; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN - 2026-03-18 13:01:37.166; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/].
# Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts'
# Checking collection health
# /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found
# solr-node1
# solr-node2
# solr-node3
# solr-node1
# solr-node2
# solr-node3
# solr-data1
# solr-data2
# solr-data3
# solrcloud-test
ok 80 allowPaths - backup # in 15745 ms
ok 81 snapshot lifecycle # in 4948 ms
ok 82 snapshot list # in 3290 ms
ok 83 snapshot describe # in 3346 ms
ok 84 start solr with ssl # in 16388 ms
ok 85 use different hostname when not checking peer-name # in 24375 ms
ok 86 start solr with ssl and auth # in 14673 ms
ok 87 start solr with client truststore and security manager # in 16720 ms
ok 88 start solr with mTLS needed # in 46274 ms
ok 89 start solr with mTLS wanted # in 34773 ms
ok 90 test keystore reload # in 59786 ms
ok 91 SOLR-11740 check 'solr stop' connection # in 15846 ms
ok 92 stop command for single port # in 18219 ms
ok 93 check stop command doesn't hang # in 219 ms
ok 94 SOLR-16976 solr starts with remote JMX enabled # in 8242 ms
ok 95 deprecated system properties converted to modern properties # in 194918 ms
ok 96 start with custom jetty options # in 8293 ms
ok 97 bootstrapping a configset # in 9244 ms
ok 98 status detects locally running solr # in 9348 ms
ok 99 status with --solr-url from user # in 7997 ms
ok 100 status with --port from user # in 8298 ms
ok 101 multiple connection options are prevented # in 867 ms
ok 102 status with invalid --solr-url from user # in 1276 ms
ok 103 status with --short format # in 195785 ms
ok 104 searching solr via locally executed streaming expression # in 1748 ms
ok 105 searching solr via remotely executed streaming expression # in 1286 ms
ok 106 variable interpolation # in 1711 ms
ok 107 searching solr without credentials fails with error # in 1134 ms
ok 108 --version returns Solr version # in 739 ms
ok 109 version as direct tool call still runs # in 722 ms
ok 110 short help # in 786 ms
ok 111 short help is inferred # in 757 ms
ok 112 long help # in 796 ms
ok 113 running subcommands with zk is prevented # in 742 ms
ok 114 listing out files # in 2154 ms
ok 115 connecting to solr via various solr urls and zk hosts # in 7933 ms
ok 116 copying files around # in 11139 ms
ok 117 upconfig # in 2590 ms
ok 118 SOLR-12429 test upconfig fails with symlink # in 1198 ms
ok 119 downconfig # in 1263 ms
ok 120 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1426 ms
ok 121 env var ZK_HOST is honored # in 3204 ms
ok 122 Cleanup # in 1848 ms
> Task :solr:packaging:integrationTests FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 336
* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 29m 13s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/e47x35i3zqb5c
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 151 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/151/
No tests ran.
Build Log:
[...truncated 952 lines...]
BUILD FAILED in 30m 35s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/limiokok3b3ae
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 150 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/150/
No tests ran.
Build Log:
[...truncated 897 lines...]
BUILD FAILED in 28m 46s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/phnk3trlrwv4c
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 149 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/149/
No tests ran.
Build Log:
[...truncated 944 lines...]
BUILD FAILED in 28m 46s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/6jtb3vrq2dvec
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 148 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/148/
No tests ran.
Build Log:
[...truncated 948 lines...]
BUILD FAILED in 28m 46s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/pi4zbdbfzabdw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 147 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/147/
No tests ran.
Build Log:
[...truncated 944 lines...]
BUILD FAILED in 28m 29s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/ovjqaeq4dhruo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 146 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/146/
No tests ran.
Build Log:
[...truncated 899 lines...]
BUILD FAILED in 29m 31s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/gfchyjidhx7wq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 145 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/145/
No tests ran.
Build Log:
[...truncated 900 lines...]
BUILD FAILED in 29m 45s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/zuiqc4hypjags
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 144 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/144/
No tests ran.
Build Log:
[...truncated 948 lines...]
BUILD FAILED in 31m 50s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/tjotrlk6l3ahu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 143 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/143/
No tests ran.
Build Log:
[...truncated 899 lines...]
BUILD FAILED in 29m 5s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/tlut5h7ltga5i
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 142 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/142/
No tests ran.
Build Log:
[...truncated 970 lines...]
BUILD FAILED in 30m 23s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/4fuukly6k6jbu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 141 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/141/
No tests ran.
Build Log:
[...truncated 903 lines...]
BUILD FAILED in 29m 26s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/qhde5zr6myfm4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 140 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/140/
No tests ran.
Build Log:
[...truncated 902 lines...]
BUILD FAILED in 28m 11s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/khsupksyymbu4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Solr » Solr-TestIntegration-10.x - Build # 139 - Still Failing!
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/139/
No tests ran.
Build Log:
[...truncated 952 lines...]
BUILD FAILED in 29m 6s
130 actionable tasks: 130 executed
Publishing Build Scan to Develocity...
https://develocity.apache.org/s/3odcbtjg62u5q
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Archiving artifacts
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any