You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Victoria Stuart (VictoriasJourney.com)" <ma...@VictoriasJourney.com> on 2022/04/26 04:15:41 UTC
Cannot post to SSL-secured core from command line
With a standalone Solr 8.11.1 instance with SSL enabled on an Arch Linux system,
[victoria@victoria solr-8.11.1]$ solr status
Found 1 Solr nodes:
Solr process 3380449 running on port 8983
INFO - 2022-04-25 20:55:03.283; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
INFO - 2022-04-25 20:55:03.286; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
{
"solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
"version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
"startTime":"2022-04-26T03:52:16.715Z",
"uptime":"0 days, 0 hours, 2 minutes, 47 seconds",
"memory":"89.3 MB (%4.4) of 2 GB"}
[victoria@victoria solr-8.11.1]$
... how can I post documents on the command line?
Do I need to provide my Solr admin username:password ( -u user;pass)?
Here I show the output without a username:password, but I got the identical
output using it.
[victoria@victoria solr-8.11.1]$ post -c pg-solr /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/virginia_lamp_thomas.html
/usr/lib/jvm/java-18-openjdk//bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dauto=yes -Dc=pg-solr -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/virginia_lamp_thomas.html
SimplePostTool version 5.0.0
Posting files to [base] url http://localhost:8983/solr/pg-solr/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file virginia_lamp_thomas.html (text/html) to [base]/extract
SimplePostTool: FATAL: IOException while posting data: java.io.IOException: Error writing request body to server
[victoria@victoria solr-8.11.1]$
If I disable SSL I can index my documents; it's a SSL issue of some type.
From the Solr Admin UI:
Core
startTime: 13 minutes ago
instanceDir: /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/pg-solr
dataDir: /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/pg-solr/data/
directory: org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/pg-solr/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a7d594d; maxCacheMB=48.0 maxMergeSizeMB=4.0)
Args
-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Dbasicauth=*** ## username:password ; user with full admin privileges ...
-Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
-Djavax.net.ssl.trustStoreType=PKCS12
-Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server
-Djetty.port=8983
-Dsolr.data.home=
-Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf
-Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
-Dsolr.install.dir=/mnt/Vancouver/apps/solr/solr-8.11.1
-Dsolr.jetty.https.port=8983
-Dsolr.jetty.inetaccess.excludes=
-Dsolr.jetty.inetaccess.includes=
-Dsolr.jetty.keystore.type=PKCS12
-Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
-Dsolr.jetty.ssl.needClientAuth=false
-Dsolr.jetty.ssl.wantClientAuth=false
-Dsolr.jetty.truststore.type=PKCS12
-Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
-Dsolr.log.dir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/logs
-Dsolr.log.muteconsole
-Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr
-Dsolr.ssl.checkPeerName=true
-Duser.timezone=UTC
-XX:+AlwaysPreTouch
-XX:+ExplicitGCInvokesConcurrent
-XX:+ParallelRefProcEnabled
-XX:+PerfDisableSharedMem
-XX:+UseG1GC
-XX:+UseLargePages
-XX:-OmitStackTraceInFastThrow
-XX:MaxGCPauseMillis=250
-XX:OnOutOfMemoryError=/mnt/Vancouver/apps/solr/solr-8.11.1/bin/oom_solr.sh 8983 /mnt/Vancouver/apps/solr/solr-8.11.1/server/logs
-Xlog:gc*:file=/mnt/Vancouver/apps/solr/solr-8.11.1/server/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
-Xms2048m
-Xmx2048m
-Xss256k
==============================================================================
Re: Cannot post to SSL-secured core from command line [solved] [addendum: passwords - character issues]
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Victoria,
On 4/26/22 21:46, Victoria Stuart (VictoriasJourney.com) wrote:
> # ----------------------------------------
> # Addendum - passwords - character issues.
> # ----------------------------------------
Hmm. You should not have had any of these issues. Can you please confirm:
1. You are saying that # does not work in a "SSL certificate password".
Do you mwan the keystore password?
Remember that you are using a bourne-shell style .sh script to configure
Solr, and that # is a special character.
SOLR_SSL_KEY_STORE_PASSWORD=secret#password
Isn't going to work as you expect. You may need to escape the # to get
the whole password:
SOLR_SSL_KEY_STORE_PASSWORD=secret\#password
You could also use quotes:
SOLR_SSL_KEY_STORE_PASSWORD="secret#password"
2. Are you saying that # does not work in an "HTTP Basic" authentication
scheme? If that's the case (and the first report I read showed a URL
with http://username:password@hostname:port/...), then the problem is
that the client is putting the authentication information into the URL
and not into the HTTP headers where they belong.
Perhaps this is a problem with one of the tools being provided by Solr
(e.g. 'post'); if so, please file a bug so it can be fixed.
-chris
> Per my earlier message [appended below], I should have mentioned that in sorting out both my Solr Basic Authentication and SSL configuration that I had been vexed by keystore and Solr passwords (I use a password generator) containing special characters (# $ etc.), that silently cause Basic Authentication / SSL connection issues.
>
> Particularly, I had issue with passwords containing the number/hash/pound character - # - echoed here:
>
> https://www.wpsolr.com/forums/topic/unable-to-connect-to-index-when-solr-authentication-and-authorization/
>
> From various documentation on the web:
>
> ** This advice was errant:
>
> https://docs.oracle.com/cd/E14571_01/install.1111/e12002/oimscrn011.htm#INOIM1372=
>
> KeyStore password : a valid password can contain 6 to 30 characters, begin with an alphabetic character,
> and use only alphanumeric characters and special characters like underscore (_), dollar ($), pound (#).
> The password must contain at least one number.
>
> ** https://getfishtank.ca/blog/updating-ssl-certificates-in-solr
>
> Point of Note: when updating to Solr certificate, there's one thing you need to be aware of: the password should not contain any special characters.
>
> It's not uncommon for SSL certs to contain special characters, but Solr doesn't like them in the format we have to work with. It should be purely alpha-numeric.
>
> If it does, during the restart you may get a message that the service failed to restart. If you get that error, this is certainly something to check.
>
> One of my original certificate p/w was apparently silently causing issues, such as the esoteric Solr console message:
>
> "... Javax.crypto.BadPaddingException:Given final block not properly padded solution ..."
>
> While user passwords generated in the Solr Admin UI may caution
>
> Password not strong enough! Must contain at least one lowercase letter, one
> uppercase letter, one digit, and one of these special characters: !@#$%^&*_-[]()
>
> As mentioned, one of my p/w contained # and so it - or the hashing/salt algorithm - resulted in silent errors (by silent I mean errors that gave no indication that the password character coding was an issue).
>
> Here is a jetty post cautioning against the use of @ in passwords:
>
> https://www.eclipse.org/lists/jetty-users/msg07410.html
>
> I would be wary of the use of non-alphanumeric "special characters" for keystore and Solr passwords. (If needed / concerned, one can increase the password length and complexity, e.g. mixed case, if concerned).
>
> * What Are Alphanumeric Characters?
> https://studyqueries.com/alphanumeric-characters/
>
> Alphanumeric characters comprise the combination of the twenty-six characters of the alphabet (from A to Z) and the numbers 0 to 9. Therefore, 1, 2, q, f, m, p, and 10 are all examples of alphanumeric characters. Symbols like *, & and @ are also considered alphanumeric characters.
>
> These characters can also be used in combination. Examples of alphanumeric characters made of the combination of special symbols, numbers, and the characters of the alphabet are &AF54hh, jjHF47, @qw99O. The characters of the alphabet can either be in lower case or upper case. The context of use determines whether or not case sensitivity is applied.
>
> * See also:
>
> ** https://stackoverflow.com/questions/34675756/http-basic-authentication-fail-with-password-with-non-iso-8859-1-characters
>
> ** https://bz.apache.org/bugzilla/show_bug.cgi?id=48985
>
> ** https://bugs.openjdk.java.net/browse/JDK-6979740
>
> ** https://issuetracker.google.com/issues/37135737 >> ... When keytool creates a KeyStore or key which is protected with a password containing non-ASCII characters, keytool may encode the password using the console's encoding or the JVM default encoding instead of using the password verbatim, in its Unicode form. ...
>
> ** https://stackoverflow.com/questions/702629/utf-8-characters-mangled-in-http-basic-auth-username >> [Ruby on Rails] "... Basic Authentication and Unicode don't mix. There is no standard for how to encode non-ASCII characters into a Basic Authentication username:password token before base64ing it. Consequently every browser does something different: ..."
>
> ** https://www.searchstax.com/docs/hc/sitecore-connect-errors/
>
> ** https://www.searchstax.com/docs/hc/unauthorized-error/
>
> ** ... etc.
>
> ==============================================================================
> ----- Original Message(s): -----
> Date: 2022 Apr 26 (Tue) 13:17
> From: "Victoria Stuart (VictoriasJourney.com)" <ma...@VictoriasJourney.com>
> To: Solr Community <us...@solr.apache.org>
> Cc: mail@VictoriasJourney.com
> Subject: Re: Cannot post to SSL-secured core from command line [solved]
>
> Thanks once again to Chris [Christopher Schultz], I have solved this issue (solution at the end of this message).
>
> For anyone who may benefit from it, I am posting a complete solution below.
>
>
> # ============================================================================
> # 1. SOLR 8.11.1 SSL CONFIGURATION
> # 2. INDEXING - TESTS (http:// | https://)
> # =====================================================================
>
> [victoria@victoria docs]$ date
> 2022-04-25 12:38:57 -0700 (PST)
>
> # ============================================================================
> # 1. SSL CERTIFICATES ...
> # =======================
>
> # ----------------------------------------
> # REMOVE OLD CERTIFICATE FROM KEYSTORE:
> # -------------------------------------
>
> # Manually delete solr-ssl* { solr-ssl.keystore.p12 | solr-ssl-cert | ... }
> # from: /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc
>
> # Delete old certificate from keystore:
> # [cacerts p/w generally defaults to: changeit]
>
> [victoria@victoria etc]$ sudo keytool -delete -trustcacerts -cacerts -storepass *** -noprompt -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert
>
> [victoria@victoria etc]$
>
>
> # ----------------------------------------
> # GENERATE NEW CERTIFICATE [solr-ssl.keystore.p12]:
> # -------------------------------------------------
>
> # NOTES:
> # 1. in this publicly-posted file: "secret" = my secret password; replace
> # "secret" with your actual password(s).
> # 2. "***" below further obfuscates keytool/other data; enter your own values.
>
> [victoria@victoria etc]$ keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 -ext SAN=DNS:localhost,IP:192.168.1.3,IP:127.0.0.1 -dname "CN=***, OU=***, O=***, L=***, ST=***, C=***"
>
> Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 9,999 days
> for: CN=***, OU=***, O=***, L=***, ST=***, C=***
>
>
> # ----------------------------------------
> # GENERATE SALTED SOLR PASSWORD HASH:
> # -----------------------------------
>
> [victoria@victoria etc]$ SALT=$(pwgen 48 -1)
> [victoria@victoria etc]$ PW='secret'
>
> [victoria@victoria etc]$ echo "hash: $(echo -n "$SALT$PW" | sha256sum -b | xxd -r -p | sha256sum -b | xxd -r -p | base64 -w 1024) $(echo -n "$SALT" | base64 -w1024)"
>
> hash: *** ***
>
> ## Note, above; <Solr hashed [obfuscated] password> <space> <salt>
>
> ## If ever needed;
>
> [victoria@victoria etc]$ sudo systemctl restart httpd
> [sudo] password for victoria:
>
> [victoria@victoria etc]$ sudo systemctl status httpd
>
> ● httpd.service - Apache Web Server
> Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
> Active: active (running) since Mon 2022-04-25 19:49:01 PDT; 6s ago
> Main PID: 3284722 (httpd)
> Tasks: 82 (limit: 38359)
> Memory: 6.6M
> CPU: 21ms
> CGroup: /system.slice/httpd.service
> ├─3284722 /usr/bin/httpd -k start -DFOREGROUND
> ├─3284723 /usr/bin/httpd -k start -DFOREGROUND
> ├─3284724 /usr/bin/httpd -k start -DFOREGROUND
> └─3284725 /usr/bin/httpd -k start -DFOREGROUND
>
> Apr 25 19:49:01 victoria systemd[1]: Started Apache Web Server.
>
>
> # ----------------------------------------
> # EXPORT CERTIFICATE FROM KEYSTORE:
> # ---------------------------------
>
> [victoria@victoria etc]$ keytool -exportcert -rfc -keystore solr-ssl.keystore.p12 -alias solr-ssl -file solr-ssl-cert
>
> Enter keystore password: ## secret
> Certificate stored in file <solr-ssl-cert>
>
> [victoria@victoria etc]$ cat solr-ssl-cert
>
> -----BEGIN CERTIFICATE-----
> ***
> -----END CERTIFICATE-----
>
>
> # ----------------------------------------
> # ADD CERTIFICATE TO JAVA TRUST STORE (cacerts):
> # ----------------------------------------------
>
> ## cacerts p/w generally defaults to: changeit
>
> [victoria@victoria etc]$ sudo keytool -import -trustcacerts -cacerts -storepass *** -noprompt -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert
>
> Certificate was added to keystore
>
> [victoria@victoria etc]$
>
>
> # ----------------------------------------
> # VERIFY KEYSTORE ADDITION:
> # -------------------------
>
> [victoria@victoria etc]$ keytool -list -cacerts | grep solr-ssl
>
> Enter keystore password: ***
> solr-ssl, Apr 25, 2022, trustedCertEntry,
>
> [victoria@victoria etc]$ keytool -list -keystore $JAVA_HOME/lib/security/cacerts | grep -i solr
>
> Warning: use -cacerts option to access cacerts keystore
> Enter keystore password: ***
> solr-ssl, Apr 25, 2022, trustedCertEntry,
>
> [victoria@victoria etc]$
>
> # ----------------------------------------
> ## Note: given the following [default installed] symlink,
>
> [victoria@victoria solr-8.11.1]$ ls -l $JAVA_HOME/lib/security/cacerts
> lrwxrwxrwx 1 root root 27 Apr 19 16:46 /usr/lib/jvm/java-18-openjdk//lib/security/cacerts -> /etc/ssl/certs/java/cacerts
>
> [victoria@victoria solr-8.11.1]$
>
> [victoria@victoria solr-8.11.1]$ keytool -list -keystore $JAVA_HOME/lib/security/cacerts | grep -i solr
> Warning: use -cacerts option to access cacerts keystore
> Enter keystore password: ***
> solr-ssl, Apr 24, 2022, trustedCertEntry,
>
> [victoria@victoria solr-8.11.1]$
>
> ## ... importing the SSL certificate to $JAVA_HOME is not needed.
>
> ## old method [not needed / not done]:
> sudo keytool -import -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.pem -alias solr-ssl -keystore $JAVA_HOME/lib/security/cacerts
>
> ## updated method [not needed / not done]:
> sudo keytool -import -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert -keystore $JAVA_HOME/lib/security/cacerts
>
> ## Check [certificate present due to symlink, above]:
>
> [victoria@victoria solr-8.11.1]$ sudo keytool -import -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert -keystore $JAVA_HOME/lib/security/cacerts
>
> [sudo] password for victoria:
> Warning: use -cacerts option to access cacerts keystore
> Enter keystore password: ## ***
> keytool error: java.lang.Exception: Certificate not imported, alias <solr-ssl> already exists
>
> [victoria@victoria solr-8.11.1]$
>
>
> # ============================================================================
> # 2. INDEX DOCUMENTS TO SSL-HARDENED SOLR
> # =======================================
>
> [victoria@victoria scripts]$ date
> 2022-04-25 12:40:29 -0700 (PST)
>
> [victoria@victoria scripts]$ apachectl configtest
> Syntax OK
>
> [victoria@victoria etc]$ pwd; ls -l
>
> /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc
> total 84K
> -rw-r--r-- 1 victoria victoria 2.0K Dec 7 17:01 jetty-gzip.xml
> -rw-r--r-- 1 victoria victoria 3.6K Dec 7 17:01 jetty-https8.xml
> -rw-r--r-- 1 victoria victoria 3.7K Dec 7 17:01 jetty-https.xml
> -rw-r--r-- 1 victoria victoria 2.7K Dec 7 17:01 jetty-http.xml
> -rw-r--r-- 1 victoria victoria 1.9K Dec 7 17:01 jetty-requestlog.xml
> -rw-r--r-- 1 victoria victoria 2.7K Apr 22 11:27 jetty-ssl.xml
> -rw-r--r-- 1 victoria victoria 12K Apr 23 12:20 jetty.xml
> -rw-r--r-- 1 victoria victoria 12K Dec 7 17:01 security.policy
> -rw-r--r-- 1 victoria victoria 1.3K Dec 7 17:01 security.properties
> -rw-r--r-- 1 victoria victoria 1.4K Apr 19 19:28 solr-ssl-cert
> -rw-r--r-- 1 victoria victoria 2.8K Apr 19 19:27 solr-ssl.keystore.p12
> -rw-r--r-- 1 victoria victoria 24K Dec 7 17:01 webdefault.xml
>
> [victoria@victoria solr]$
>
>
> # ----------------------------------------
> # PATHS:
> # ------
>
> [victoria@victoria solr]$ echo $SOLR_JAVA_HOME/
> /usr/lib/jvm/java-18-openjdk//
>
> [victoria@victoria solr]$ echo $PATH | sed 's/:/\n/g' | grep solr
> /mnt/Vancouver/apps/solr/solr-8.11.1/bin/
>
> [victoria@victoria solr]$
>
>
> # ----------------------------------------
> # solr.in.sh :
> # ------------
>
> ## Note: basic authentication allows access to SSL-protected Solr from the console / command-line.
>
> SOLR_SSL_ENABLED=true
>
> SOLR_SSL_KEY_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
> SOLR_SSL_KEY_STORE_PASSWORD=secret
> SOLR_SSL_KEY_STORE_TYPE=PKCS12
>
> SOLR_SSL_TRUST_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
> SOLR_SSL_TRUST_STORE_PASSWORD=secret
> SOLR_SSL_TRUST_STORE_TYPE=PKCS12
>
> SOLR_AUTH_TYPE="basic"
> SOLR_AUTHENTICATION_OPTS="-Dbasicauth=pg-solr-admin:secret"
>
> SOLR_SSL_NEED_CLIENT_AUTH=false
> SOLR_SSL_WANT_CLIENT_AUTH=false
>
> SOLR_SSL_CLIENT_HOSTNAME_VERIFICATION=true
>
> SOLR_SSL_CHECK_PEER_NAME=true
>
>
> # ----------------------------------------
> # security.json [note: p/w obfuscated (removed), below]:
> # ------------------------------------------------------
>
> {
> "authentication":{
> "blockUnknown":true,
> "class":"solr.BasicAuthPlugin",
> "credentials":{"pg-solr-admin":"***"},
> "realm":"Persagen",
> "forwardCredentials":false},
> "authorization":{
> "class":"solr.RuleBasedAuthorizationPlugin",
> "permissions":[
> {
> "name":"security-edit",
> "role":"admin",
> "index":1},
> {
> "name":"schema-read",
> "role":["admin"],
> "index":2},
> {
> "name":"security-read",
> "role":["admin"],
> "index":3},
> {
> "name":"all",
> "role":["admin"],
> "index":4}],
> "user-role":{"pg-solr-admin":["admin"]},
> "":{"v":0}}}
>
>
> # ----------------------------------------
> # JAVA VERSIONS:
> # ---------------
>
> [victoria@victoria solr]$ which java
> /usr/bin/java
>
> [victoria@victoria solr]$ whereis java
> java: /usr/bin/java /usr/share/java /usr/lib/jvm/java-18-openjdk/bin/java /usr/share/man/man1/java.1.gz
>
> [victoria@victoria solr]$
>
>
> # ----------------------------------------
> # SOLR INDEXING (old, for reference; note: http://...):
> # -----------------------------------------------------
>
> /usr/lib/jvm/java-8-openjdk/jre//bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.7.0/dist/solr-core-8.7.0.jar -Dauto=yes -Dc=core0 -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/programming/datasci/solr/test/d1.html /mnt/Vancouver/programming/datasci/solr/test/d2.html /mnt/Vancouver/programming/datasci/solr/test/d3.html /mnt/Vancouver/programming/datasci/solr/test/d4.html
>
> /usr/lib/jvm/java-8-openjdk/jre//bin/java
> -classpath /mnt/Vancouver/apps/solr/solr-8.7.0/dist/solr-core-8.7.0.jar
> -Dauto=yes
> -Dc=core0
> -Ddata=files
> org.apache.solr.util.SimplePostTool
> /mnt/Vancouver/programming/datasci/solr/test/d1.html
> /mnt/Vancouver/programming/datasci/solr/test/d2.html
> /mnt/Vancouver/programming/datasci/solr/test/d3.html
> /mnt/Vancouver/programming/datasci/solr/test/d4.html
>
>
> # ----------------------------------------
> # SOLR ADMIN UI (HINTS FOR PARAMETRIZATION):
> # https://localhost:8983/solr/#/
> # ------------------------------------------
>
> -DSTOP.KEY=solrrocks
> -DSTOP.PORT=7983
> -Dbasicauth=pg-solr-admin:secret
> -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
> -Djavax.net.ssl.keyStoreType=PKCS12
> [ ... snip ... ]
> -Xms2048m
> -Xmx2048m
> -Xss256k
>
>
> ## NOTE: I want to run multiple Solr cores, and I am indexing HTML pages,
> ## metadata using bespoke { solrconfig.xml | schema.xml } files.
> ## For simplicity/testing, I temporarily moved to those *.xml files to:
> ## /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/
>
>
> # ----------------------------------------------------------------------------
> # DELETE OLD CONTENT (SOLR ADMIN UI; XML DOCUMENT TYPE):
> # ------------------------------------------------------
>
> <delete><query>*:*</query></delete>
>
>
> # ----------------------------------------
> # COMMAND-LINE POST TO SOLR:
> # --------------------------
>
> ## Template:
>
> /usr/lib/jvm/java-18-openjdk/bin/java \
> -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar \
> -Dbasicauth=pg-solr-admin:secret \
> -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ \
> -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Djavax.net.ssl.keyStoreType=PKCS12 \
> -Djavax.net.ssl.keyStorePassword=secret \
> -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Djavax.net.ssl.trustStoreType=PKCS12 \
> -Djavax.net.ssl.trustStorePassword=secret \
> -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server \
> -Djetty.port=8983 \
> -Dsolr.jetty.https.port=8983 \
> -Dsolr.jetty.keystore.type=PKCS12 \
> -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Dsolr.jetty.ssl.needClientAuth=false \
> -Dsolr.jetty.ssl.wantClientAuth=false \
> -Dsolr.jetty.truststore.type=PKCS12 \
> -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr \
> -Dauto=yes \
> -Dc=pg-solr \
> -Durl=https://localhost:8983/solr/pg-solr/update \
> -Ddata=files \
> org.apache.solr.util.SimplePostTool \
> /mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
>
>
> # ----------------------------------------
> # INDEX DOCUMENT TO "pg-solr" CORE [SUCCESS! 😀]
> # ----------------------------------------------
>
> [victoria@victoria solr]$ solr status
>
> Found 1 Solr nodes:
>
> Solr process 272444 running on port 8983
> INFO - 2022-04-26 10:26:55.684; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
> INFO - 2022-04-26 10:26:55.686; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
> {
> "solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
> "version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
> "startTime":"2022-04-26T16:27:45.310Z",
> "uptime":"0 days, 0 hours, 59 minutes, 10 seconds",
> "memory":"251.5 MB (%12.3) of 2 GB"}
>
> [victoria@victoria solr]$
>
> ## POST HTML DOCUMENT:
>
> [victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java \
> -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar \
> -Dbasicauth=pg-solr-admin:secret \
> -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ \
> -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Djavax.net.ssl.keyStoreType=PKCS12 \
> -Djavax.net.ssl.keyStorePassword=secret \
> -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Djavax.net.ssl.trustStoreType=PKCS12 \
> -Djavax.net.ssl.trustStorePassword=secret \
> -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server \
> -Djetty.port=8983 \
> -Dsolr.jetty.https.port=8983 \
> -Dsolr.jetty.keystore.type=PKCS12 \
> -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Dsolr.jetty.ssl.needClientAuth=false \
> -Dsolr.jetty.ssl.wantClientAuth=false \
> -Dsolr.jetty.truststore.type=PKCS12 \
> -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr \
> -Dauto=yes \
> -Dc=pg-solr \
> -Durl=https://localhost:8983/solr/pg-solr/update \
> -Ddata=files \
> org.apache.solr.util.SimplePostTool \
> /mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
>
> SimplePostTool version 5.0.0
> Basic Authentication enabled, user=pg-solr-admin
> Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
> Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
> POSTing file jeff_bezos.html (text/html) to [base]/extract
> 1 files indexed.
> COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
> Time spent: 0:00:01.214
>
> # ----------------------------------------
> # REPEAT (TEST: DID NOT DUPLICATE EXISTING INDEXED ENTRY 👍):
> # -----------------------------------------------------------
>
> [victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
>
> SimplePostTool version 5.0.0
> Basic Authentication enabled, user=pg-solr-admin
> Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
> Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
> POSTing file jeff_bezos.html (text/html) to [base]/extract
> 1 files indexed.
> COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
> Time spent: 0:00:02.336
>
> [victoria@victoria solr]$
>
>
> # ----------------------------------------------------------------------------
> # CREATE NEW SOLR CORE:
> # ---------------------
>
> [victoria@victoria solr]$ solr create -c pg-solr-demo
>
> WARNING: Using _default configset with data driven schema functionality. NOT RECOMMENDED for production use.
> To turn off: bin/solr config -c pg-solr-demo -p 8983 -action set-user-property -property update.autoCreateFields -value false
> INFO - 2022-04-26 11:22:02.802; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
> INFO - 2022-04-26 11:22:02.804; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
>
> Created new core 'pg-solr-demo'
>
> [victoria@victoria solr]$ solr restart
>
> Sending stop command to Solr running on port 8983 ... waiting up to 45 seconds to allow Jetty process 272444 to stop gracefully.
> Waiting up to 45 seconds to see Solr running on port 8983 [|]
> Started Solr server on port 8983 (pid=446422). Happy searching!
>
> [victoria@victoria solr]$ solr status
>
> Found 1 Solr nodes:
>
> Solr process 446422 running on port 8983
> INFO - 2022-04-26 11:22:25.221; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
> INFO - 2022-04-26 11:22:25.223; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
> {
> "solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
> "version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
> "startTime":"2022-04-26T18:22:12.715Z",
> "uptime":"0 days, 0 hours, 0 minutes, 13 seconds",
> "memory":"54.2 MB (%2.6) of 2 GB"}
>
> [victoria@victoria solr]$
>
> # ----------------------------------------------------------------------------
> # INDEX NEW DOCUMENT TO ORIGINAL CORE [pg-solr]:
> # ----------------------------------------------
>
> [victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/greg_abbott.html
>
> SimplePostTool version 5.0.0
> Basic Authentication enabled, user=pg-solr-admin
> Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
> Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
> POSTing file greg_abbott.html (text/html) to [base]/extract
> 1 files indexed.
> COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
> Time spent: 0:00:02.287
>
> [victoria@victoria solr]$
>
>
> # ----------------------------------------------------------------------------
> # INDEX NEW DOCUMENT TO NEW CORE [pg-solr-demo]:
> # ----------------------------------------------
>
> [victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr-demo/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/neofascism.html
>
> SimplePostTool version 5.0.0
> Basic Authentication enabled, user=pg-solr-admin
> Posting files to [base] url https://localhost:8983/solr/pg-solr-demo/update...
> Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
> POSTing file neofascism.html (text/html) to [base]/extract
> 1 files indexed.
> COMMITting Solr index changes to https://localhost:8983/solr/pg-solr-demo/update...
> Time spent: 0:00:01.715
>
> [victoria@victoria solr]$
>
> # ----------------------------------------------------------------------------
>
> ## Q.E.D 😅
>
> # ============================================================================
> ==============================================================================
Re: Cannot post to SSL-secured core from command line [solved] [addendum: passwords - character issues]
Posted by "Victoria Stuart (VictoriasJourney.com)" <ma...@VictoriasJourney.com>.
# ----------------------------------------
# Addendum - passwords - character issues.
# ----------------------------------------
Per my earlier message [appended below], I should have mentioned that in sorting out both my Solr Basic Authentication and SSL configuration that I had been vexed by keystore and Solr passwords (I use a password generator) containing special characters (# $ etc.), that silently cause Basic Authentication / SSL connection issues.
Particularly, I had issue with passwords containing the number/hash/pound character - # - echoed here:
https://www.wpsolr.com/forums/topic/unable-to-connect-to-index-when-solr-authentication-and-authorization/
From various documentation on the web:
** This advice was errant:
https://docs.oracle.com/cd/E14571_01/install.1111/e12002/oimscrn011.htm#INOIM1372=
KeyStore password : a valid password can contain 6 to 30 characters, begin with an alphabetic character,
and use only alphanumeric characters and special characters like underscore (_), dollar ($), pound (#).
The password must contain at least one number.
** https://getfishtank.ca/blog/updating-ssl-certificates-in-solr
Point of Note: when updating to Solr certificate, there's one thing you need to be aware of: the password should not contain any special characters.
It's not uncommon for SSL certs to contain special characters, but Solr doesn't like them in the format we have to work with. It should be purely alpha-numeric.
If it does, during the restart you may get a message that the service failed to restart. If you get that error, this is certainly something to check.
One of my original certificate p/w was apparently silently causing issues, such as the esoteric Solr console message:
"... Javax.crypto.BadPaddingException:Given final block not properly padded solution ..."
While user passwords generated in the Solr Admin UI may caution
Password not strong enough! Must contain at least one lowercase letter, one
uppercase letter, one digit, and one of these special characters: !@#$%^&*_-[]()
As mentioned, one of my p/w contained # and so it - or the hashing/salt algorithm - resulted in silent errors (by silent I mean errors that gave no indication that the password character coding was an issue).
Here is a jetty post cautioning against the use of @ in passwords:
https://www.eclipse.org/lists/jetty-users/msg07410.html
I would be wary of the use of non-alphanumeric "special characters" for keystore and Solr passwords. (If needed / concerned, one can increase the password length and complexity, e.g. mixed case, if concerned).
* What Are Alphanumeric Characters?
https://studyqueries.com/alphanumeric-characters/
Alphanumeric characters comprise the combination of the twenty-six characters of the alphabet (from A to Z) and the numbers 0 to 9. Therefore, 1, 2, q, f, m, p, and 10 are all examples of alphanumeric characters. Symbols like *, & and @ are also considered alphanumeric characters.
These characters can also be used in combination. Examples of alphanumeric characters made of the combination of special symbols, numbers, and the characters of the alphabet are &AF54hh, jjHF47, @qw99O. The characters of the alphabet can either be in lower case or upper case. The context of use determines whether or not case sensitivity is applied.
* See also:
** https://stackoverflow.com/questions/34675756/http-basic-authentication-fail-with-password-with-non-iso-8859-1-characters
** https://bz.apache.org/bugzilla/show_bug.cgi?id=48985
** https://bugs.openjdk.java.net/browse/JDK-6979740
** https://issuetracker.google.com/issues/37135737 >> ... When keytool creates a KeyStore or key which is protected with a password containing non-ASCII characters, keytool may encode the password using the console's encoding or the JVM default encoding instead of using the password verbatim, in its Unicode form. ...
** https://stackoverflow.com/questions/702629/utf-8-characters-mangled-in-http-basic-auth-username >> [Ruby on Rails] "... Basic Authentication and Unicode don't mix. There is no standard for how to encode non-ASCII characters into a Basic Authentication username:password token before base64ing it. Consequently every browser does something different: ..."
** https://www.searchstax.com/docs/hc/sitecore-connect-errors/
** https://www.searchstax.com/docs/hc/unauthorized-error/
** ... etc.
==============================================================================
----- Original Message(s): -----
Date: 2022 Apr 26 (Tue) 13:17
From: "Victoria Stuart (VictoriasJourney.com)" <ma...@VictoriasJourney.com>
To: Solr Community <us...@solr.apache.org>
Cc: mail@VictoriasJourney.com
Subject: Re: Cannot post to SSL-secured core from command line [solved]
Thanks once again to Chris [Christopher Schultz], I have solved this issue (solution at the end of this message).
For anyone who may benefit from it, I am posting a complete solution below.
# ============================================================================
# 1. SOLR 8.11.1 SSL CONFIGURATION
# 2. INDEXING - TESTS (http:// | https://)
# =====================================================================
[victoria@victoria docs]$ date
2022-04-25 12:38:57 -0700 (PST)
# ============================================================================
# 1. SSL CERTIFICATES ...
# =======================
# ----------------------------------------
# REMOVE OLD CERTIFICATE FROM KEYSTORE:
# -------------------------------------
# Manually delete solr-ssl* { solr-ssl.keystore.p12 | solr-ssl-cert | ... }
# from: /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc
# Delete old certificate from keystore:
# [cacerts p/w generally defaults to: changeit]
[victoria@victoria etc]$ sudo keytool -delete -trustcacerts -cacerts -storepass *** -noprompt -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert
[victoria@victoria etc]$
# ----------------------------------------
# GENERATE NEW CERTIFICATE [solr-ssl.keystore.p12]:
# -------------------------------------------------
# NOTES:
# 1. in this publicly-posted file: "secret" = my secret password; replace
# "secret" with your actual password(s).
# 2. "***" below further obfuscates keytool/other data; enter your own values.
[victoria@victoria etc]$ keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 -ext SAN=DNS:localhost,IP:192.168.1.3,IP:127.0.0.1 -dname "CN=***, OU=***, O=***, L=***, ST=***, C=***"
Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 9,999 days
for: CN=***, OU=***, O=***, L=***, ST=***, C=***
# ----------------------------------------
# GENERATE SALTED SOLR PASSWORD HASH:
# -----------------------------------
[victoria@victoria etc]$ SALT=$(pwgen 48 -1)
[victoria@victoria etc]$ PW='secret'
[victoria@victoria etc]$ echo "hash: $(echo -n "$SALT$PW" | sha256sum -b | xxd -r -p | sha256sum -b | xxd -r -p | base64 -w 1024) $(echo -n "$SALT" | base64 -w1024)"
hash: *** ***
## Note, above; <Solr hashed [obfuscated] password> <space> <salt>
## If ever needed;
[victoria@victoria etc]$ sudo systemctl restart httpd
[sudo] password for victoria:
[victoria@victoria etc]$ sudo systemctl status httpd
● httpd.service - Apache Web Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-04-25 19:49:01 PDT; 6s ago
Main PID: 3284722 (httpd)
Tasks: 82 (limit: 38359)
Memory: 6.6M
CPU: 21ms
CGroup: /system.slice/httpd.service
├─3284722 /usr/bin/httpd -k start -DFOREGROUND
├─3284723 /usr/bin/httpd -k start -DFOREGROUND
├─3284724 /usr/bin/httpd -k start -DFOREGROUND
└─3284725 /usr/bin/httpd -k start -DFOREGROUND
Apr 25 19:49:01 victoria systemd[1]: Started Apache Web Server.
# ----------------------------------------
# EXPORT CERTIFICATE FROM KEYSTORE:
# ---------------------------------
[victoria@victoria etc]$ keytool -exportcert -rfc -keystore solr-ssl.keystore.p12 -alias solr-ssl -file solr-ssl-cert
Enter keystore password: ## secret
Certificate stored in file <solr-ssl-cert>
[victoria@victoria etc]$ cat solr-ssl-cert
-----BEGIN CERTIFICATE-----
***
-----END CERTIFICATE-----
# ----------------------------------------
# ADD CERTIFICATE TO JAVA TRUST STORE (cacerts):
# ----------------------------------------------
## cacerts p/w generally defaults to: changeit
[victoria@victoria etc]$ sudo keytool -import -trustcacerts -cacerts -storepass *** -noprompt -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert
Certificate was added to keystore
[victoria@victoria etc]$
# ----------------------------------------
# VERIFY KEYSTORE ADDITION:
# -------------------------
[victoria@victoria etc]$ keytool -list -cacerts | grep solr-ssl
Enter keystore password: ***
solr-ssl, Apr 25, 2022, trustedCertEntry,
[victoria@victoria etc]$ keytool -list -keystore $JAVA_HOME/lib/security/cacerts | grep -i solr
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: ***
solr-ssl, Apr 25, 2022, trustedCertEntry,
[victoria@victoria etc]$
# ----------------------------------------
## Note: given the following [default installed] symlink,
[victoria@victoria solr-8.11.1]$ ls -l $JAVA_HOME/lib/security/cacerts
lrwxrwxrwx 1 root root 27 Apr 19 16:46 /usr/lib/jvm/java-18-openjdk//lib/security/cacerts -> /etc/ssl/certs/java/cacerts
[victoria@victoria solr-8.11.1]$
[victoria@victoria solr-8.11.1]$ keytool -list -keystore $JAVA_HOME/lib/security/cacerts | grep -i solr
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: ***
solr-ssl, Apr 24, 2022, trustedCertEntry,
[victoria@victoria solr-8.11.1]$
## ... importing the SSL certificate to $JAVA_HOME is not needed.
## old method [not needed / not done]:
sudo keytool -import -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.pem -alias solr-ssl -keystore $JAVA_HOME/lib/security/cacerts
## updated method [not needed / not done]:
sudo keytool -import -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert -keystore $JAVA_HOME/lib/security/cacerts
## Check [certificate present due to symlink, above]:
[victoria@victoria solr-8.11.1]$ sudo keytool -import -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert -keystore $JAVA_HOME/lib/security/cacerts
[sudo] password for victoria:
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: ## ***
keytool error: java.lang.Exception: Certificate not imported, alias <solr-ssl> already exists
[victoria@victoria solr-8.11.1]$
# ============================================================================
# 2. INDEX DOCUMENTS TO SSL-HARDENED SOLR
# =======================================
[victoria@victoria scripts]$ date
2022-04-25 12:40:29 -0700 (PST)
[victoria@victoria scripts]$ apachectl configtest
Syntax OK
[victoria@victoria etc]$ pwd; ls -l
/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc
total 84K
-rw-r--r-- 1 victoria victoria 2.0K Dec 7 17:01 jetty-gzip.xml
-rw-r--r-- 1 victoria victoria 3.6K Dec 7 17:01 jetty-https8.xml
-rw-r--r-- 1 victoria victoria 3.7K Dec 7 17:01 jetty-https.xml
-rw-r--r-- 1 victoria victoria 2.7K Dec 7 17:01 jetty-http.xml
-rw-r--r-- 1 victoria victoria 1.9K Dec 7 17:01 jetty-requestlog.xml
-rw-r--r-- 1 victoria victoria 2.7K Apr 22 11:27 jetty-ssl.xml
-rw-r--r-- 1 victoria victoria 12K Apr 23 12:20 jetty.xml
-rw-r--r-- 1 victoria victoria 12K Dec 7 17:01 security.policy
-rw-r--r-- 1 victoria victoria 1.3K Dec 7 17:01 security.properties
-rw-r--r-- 1 victoria victoria 1.4K Apr 19 19:28 solr-ssl-cert
-rw-r--r-- 1 victoria victoria 2.8K Apr 19 19:27 solr-ssl.keystore.p12
-rw-r--r-- 1 victoria victoria 24K Dec 7 17:01 webdefault.xml
[victoria@victoria solr]$
# ----------------------------------------
# PATHS:
# ------
[victoria@victoria solr]$ echo $SOLR_JAVA_HOME/
/usr/lib/jvm/java-18-openjdk//
[victoria@victoria solr]$ echo $PATH | sed 's/:/\n/g' | grep solr
/mnt/Vancouver/apps/solr/solr-8.11.1/bin/
[victoria@victoria solr]$
# ----------------------------------------
# solr.in.sh :
# ------------
## Note: basic authentication allows access to SSL-protected Solr from the console / command-line.
SOLR_SSL_ENABLED=true
SOLR_SSL_KEY_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
SOLR_SSL_KEY_STORE_PASSWORD=secret
SOLR_SSL_KEY_STORE_TYPE=PKCS12
SOLR_SSL_TRUST_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
SOLR_SSL_TRUST_STORE_PASSWORD=secret
SOLR_SSL_TRUST_STORE_TYPE=PKCS12
SOLR_AUTH_TYPE="basic"
SOLR_AUTHENTICATION_OPTS="-Dbasicauth=pg-solr-admin:secret"
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false
SOLR_SSL_CLIENT_HOSTNAME_VERIFICATION=true
SOLR_SSL_CHECK_PEER_NAME=true
# ----------------------------------------
# security.json [note: p/w obfuscated (removed), below]:
# ------------------------------------------------------
{
"authentication":{
"blockUnknown":true,
"class":"solr.BasicAuthPlugin",
"credentials":{"pg-solr-admin":"***"},
"realm":"Persagen",
"forwardCredentials":false},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[
{
"name":"security-edit",
"role":"admin",
"index":1},
{
"name":"schema-read",
"role":["admin"],
"index":2},
{
"name":"security-read",
"role":["admin"],
"index":3},
{
"name":"all",
"role":["admin"],
"index":4}],
"user-role":{"pg-solr-admin":["admin"]},
"":{"v":0}}}
# ----------------------------------------
# JAVA VERSIONS:
# ---------------
[victoria@victoria solr]$ which java
/usr/bin/java
[victoria@victoria solr]$ whereis java
java: /usr/bin/java /usr/share/java /usr/lib/jvm/java-18-openjdk/bin/java /usr/share/man/man1/java.1.gz
[victoria@victoria solr]$
# ----------------------------------------
# SOLR INDEXING (old, for reference; note: http://...):
# -----------------------------------------------------
/usr/lib/jvm/java-8-openjdk/jre//bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.7.0/dist/solr-core-8.7.0.jar -Dauto=yes -Dc=core0 -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/programming/datasci/solr/test/d1.html /mnt/Vancouver/programming/datasci/solr/test/d2.html /mnt/Vancouver/programming/datasci/solr/test/d3.html /mnt/Vancouver/programming/datasci/solr/test/d4.html
/usr/lib/jvm/java-8-openjdk/jre//bin/java
-classpath /mnt/Vancouver/apps/solr/solr-8.7.0/dist/solr-core-8.7.0.jar
-Dauto=yes
-Dc=core0
-Ddata=files
org.apache.solr.util.SimplePostTool
/mnt/Vancouver/programming/datasci/solr/test/d1.html
/mnt/Vancouver/programming/datasci/solr/test/d2.html
/mnt/Vancouver/programming/datasci/solr/test/d3.html
/mnt/Vancouver/programming/datasci/solr/test/d4.html
# ----------------------------------------
# SOLR ADMIN UI (HINTS FOR PARAMETRIZATION):
# https://localhost:8983/solr/#/
# ------------------------------------------
-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Dbasicauth=pg-solr-admin:secret
-Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
-Djavax.net.ssl.keyStoreType=PKCS12
[ ... snip ... ]
-Xms2048m
-Xmx2048m
-Xss256k
## NOTE: I want to run multiple Solr cores, and I am indexing HTML pages,
## metadata using bespoke { solrconfig.xml | schema.xml } files.
## For simplicity/testing, I temporarily moved to those *.xml files to:
## /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/
# ----------------------------------------------------------------------------
# DELETE OLD CONTENT (SOLR ADMIN UI; XML DOCUMENT TYPE):
# ------------------------------------------------------
<delete><query>*:*</query></delete>
# ----------------------------------------
# COMMAND-LINE POST TO SOLR:
# --------------------------
## Template:
/usr/lib/jvm/java-18-openjdk/bin/java \
-classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar \
-Dbasicauth=pg-solr-admin:secret \
-Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ \
-Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.keyStoreType=PKCS12 \
-Djavax.net.ssl.keyStorePassword=secret \
-Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.trustStoreType=PKCS12 \
-Djavax.net.ssl.trustStorePassword=secret \
-Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server \
-Djetty.port=8983 \
-Dsolr.jetty.https.port=8983 \
-Dsolr.jetty.keystore.type=PKCS12 \
-Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.jetty.ssl.needClientAuth=false \
-Dsolr.jetty.ssl.wantClientAuth=false \
-Dsolr.jetty.truststore.type=PKCS12 \
-Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr \
-Dauto=yes \
-Dc=pg-solr \
-Durl=https://localhost:8983/solr/pg-solr/update \
-Ddata=files \
org.apache.solr.util.SimplePostTool \
/mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
# ----------------------------------------
# INDEX DOCUMENT TO "pg-solr" CORE [SUCCESS! 😀]
# ----------------------------------------------
[victoria@victoria solr]$ solr status
Found 1 Solr nodes:
Solr process 272444 running on port 8983
INFO - 2022-04-26 10:26:55.684; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
INFO - 2022-04-26 10:26:55.686; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
{
"solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
"version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
"startTime":"2022-04-26T16:27:45.310Z",
"uptime":"0 days, 0 hours, 59 minutes, 10 seconds",
"memory":"251.5 MB (%12.3) of 2 GB"}
[victoria@victoria solr]$
## POST HTML DOCUMENT:
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java \
-classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar \
-Dbasicauth=pg-solr-admin:secret \
-Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ \
-Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.keyStoreType=PKCS12 \
-Djavax.net.ssl.keyStorePassword=secret \
-Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.trustStoreType=PKCS12 \
-Djavax.net.ssl.trustStorePassword=secret \
-Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server \
-Djetty.port=8983 \
-Dsolr.jetty.https.port=8983 \
-Dsolr.jetty.keystore.type=PKCS12 \
-Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.jetty.ssl.needClientAuth=false \
-Dsolr.jetty.ssl.wantClientAuth=false \
-Dsolr.jetty.truststore.type=PKCS12 \
-Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr \
-Dauto=yes \
-Dc=pg-solr \
-Durl=https://localhost:8983/solr/pg-solr/update \
-Ddata=files \
org.apache.solr.util.SimplePostTool \
/mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file jeff_bezos.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
Time spent: 0:00:01.214
# ----------------------------------------
# REPEAT (TEST: DID NOT DUPLICATE EXISTING INDEXED ENTRY 👍):
# -----------------------------------------------------------
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file jeff_bezos.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
Time spent: 0:00:02.336
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
# CREATE NEW SOLR CORE:
# ---------------------
[victoria@victoria solr]$ solr create -c pg-solr-demo
WARNING: Using _default configset with data driven schema functionality. NOT RECOMMENDED for production use.
To turn off: bin/solr config -c pg-solr-demo -p 8983 -action set-user-property -property update.autoCreateFields -value false
INFO - 2022-04-26 11:22:02.802; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
INFO - 2022-04-26 11:22:02.804; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
Created new core 'pg-solr-demo'
[victoria@victoria solr]$ solr restart
Sending stop command to Solr running on port 8983 ... waiting up to 45 seconds to allow Jetty process 272444 to stop gracefully.
Waiting up to 45 seconds to see Solr running on port 8983 [|]
Started Solr server on port 8983 (pid=446422). Happy searching!
[victoria@victoria solr]$ solr status
Found 1 Solr nodes:
Solr process 446422 running on port 8983
INFO - 2022-04-26 11:22:25.221; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
INFO - 2022-04-26 11:22:25.223; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
{
"solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
"version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
"startTime":"2022-04-26T18:22:12.715Z",
"uptime":"0 days, 0 hours, 0 minutes, 13 seconds",
"memory":"54.2 MB (%2.6) of 2 GB"}
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
# INDEX NEW DOCUMENT TO ORIGINAL CORE [pg-solr]:
# ----------------------------------------------
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/greg_abbott.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file greg_abbott.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
Time spent: 0:00:02.287
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
# INDEX NEW DOCUMENT TO NEW CORE [pg-solr-demo]:
# ----------------------------------------------
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr-demo/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/neofascism.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr-demo/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file neofascism.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr-demo/update...
Time spent: 0:00:01.715
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
## Q.E.D 😅
# ============================================================================
==============================================================================
Re: Cannot post to SSL-secured core from command line [solved]
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Victoria,
On 4/26/22 16:17, Victoria Stuart (VictoriasJourney.com) wrote:
>
> [snip]
>
> [victoria@victoria etc]$ sudo systemctl restart httpd
> [sudo] password for victoria:
I think this httpd restart/status are not relevant, no?
> # ----------------------------------------
> # ADD CERTIFICATE TO JAVA TRUST STORE (cacerts):
> # ----------------------------------------------
>
> ## cacerts p/w generally defaults to: changeit
>
> [victoria@victoria etc]$ sudo keytool -import -trustcacerts -cacerts -storepass *** -noprompt -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert
>
> Certificate was added to keystore
I would highly recommend *against* modifying the platform's cacerts
trust store. It should be possible to use a specific trust store for any
client who needs to access your Solr server.
> # ============================================================================
> # 2. INDEX DOCUMENTS TO SSL-HARDENED SOLR
> # =======================================
>
> [snip]
>
> # ----------------------------------------
> # solr.in.sh :
> # ------------
>
> ## Note: basic authentication allows access to SSL-protected Solr from the console / command-line.
>
> SOLR_SSL_ENABLED=true
>
> SOLR_SSL_KEY_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
> SOLR_SSL_KEY_STORE_PASSWORD=secret
> SOLR_SSL_KEY_STORE_TYPE=PKCS12
>
> SOLR_SSL_TRUST_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
> SOLR_SSL_TRUST_STORE_PASSWORD=secret
> SOLR_SSL_TRUST_STORE_TYPE=PKCS12
>
> SOLR_AUTH_TYPE="basic"
> SOLR_AUTHENTICATION_OPTS="-Dbasicauth=pg-solr-admin:secret"
>
> SOLR_SSL_NEED_CLIENT_AUTH=false
> SOLR_SSL_WANT_CLIENT_AUTH=false
Hmm I could have sworn you were using mutual-TLS. Maybe not.
> # ----------------------------------------
> # SOLR INDEXING (old, for reference; note: http://...):
> # -----------------------------------------------------
>
> /usr/lib/jvm/java-8-openjdk/jre//bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.7.0/dist/solr-core-8.7.0.jar -Dauto=yes -Dc=core0 -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/programming/datasci/solr/test/d1.html /mnt/Vancouver/programming/datasci/solr/test/d2.html /mnt/Vancouver/programming/datasci/solr/test/d3.html /mnt/Vancouver/programming/datasci/solr/test/d4.html
If you add:
-Djavax.net.ssl.trustStore=[path to trust store]
-Djavax.net.ssl.trustStorePassword=[password]
-Djavax.net.ssl.trustStoreType=[type]
... then you should not have to modify the platform's cacerts trust store.
> /usr/lib/jvm/java-18-openjdk/bin/java \
> -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar \
> -Dbasicauth=pg-solr-admin:secret \
> -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ \
> -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Djavax.net.ssl.keyStoreType=PKCS12 \
> -Djavax.net.ssl.keyStorePassword=secret \
> -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
> -Djavax.net.ssl.trustStoreType=PKCS12 \
> -Djavax.net.ssl.trustStorePassword=secret \
Yes, just like the above.
-chris
Re: Cannot post to SSL-secured core from command line [solved]
Posted by "Victoria Stuart (VictoriasJourney.com)" <ma...@VictoriasJourney.com>.
Thanks once again to Chris [Christopher Schultz], I have solved this issue (solution at the end of this message).
For anyone who may benefit from it, I am posting a complete solution below.
# ============================================================================
# 1. SOLR 8.11.1 SSL CONFIGURATION
# 2. INDEXING - TESTS (http:// | https://)
# =====================================================================
[victoria@victoria docs]$ date
2022-04-25 12:38:57 -0700 (PST)
# ============================================================================
# 1. SSL CERTIFICATES ...
# =======================
# ----------------------------------------
# REMOVE OLD CERTIFICATE FROM KEYSTORE:
# -------------------------------------
# Manually delete solr-ssl* { solr-ssl.keystore.p12 | solr-ssl-cert | ... }
# from: /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc
# Delete old certificate from keystore:
# [cacerts p/w generally defaults to: changeit]
[victoria@victoria etc]$ sudo keytool -delete -trustcacerts -cacerts -storepass *** -noprompt -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert
[victoria@victoria etc]$
# ----------------------------------------
# GENERATE NEW CERTIFICATE [solr-ssl.keystore.p12]:
# -------------------------------------------------
# NOTES:
# 1. in this publicly-posted file: "secret" = my secret password; replace
# "secret" with your actual password(s).
# 2. "***" below further obfuscates keytool/other data; enter your own values.
[victoria@victoria etc]$ keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 -ext SAN=DNS:localhost,IP:192.168.1.3,IP:127.0.0.1 -dname "CN=***, OU=***, O=***, L=***, ST=***, C=***"
Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 9,999 days
for: CN=***, OU=***, O=***, L=***, ST=***, C=***
# ----------------------------------------
# GENERATE SALTED SOLR PASSWORD HASH:
# -----------------------------------
[victoria@victoria etc]$ SALT=$(pwgen 48 -1)
[victoria@victoria etc]$ PW='secret'
[victoria@victoria etc]$ echo "hash: $(echo -n "$SALT$PW" | sha256sum -b | xxd -r -p | sha256sum -b | xxd -r -p | base64 -w 1024) $(echo -n "$SALT" | base64 -w1024)"
hash: *** ***
## Note, above; <Solr hashed [obfuscated] password> <space> <salt>
## If ever needed;
[victoria@victoria etc]$ sudo systemctl restart httpd
[sudo] password for victoria:
[victoria@victoria etc]$ sudo systemctl status httpd
● httpd.service - Apache Web Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-04-25 19:49:01 PDT; 6s ago
Main PID: 3284722 (httpd)
Tasks: 82 (limit: 38359)
Memory: 6.6M
CPU: 21ms
CGroup: /system.slice/httpd.service
├─3284722 /usr/bin/httpd -k start -DFOREGROUND
├─3284723 /usr/bin/httpd -k start -DFOREGROUND
├─3284724 /usr/bin/httpd -k start -DFOREGROUND
└─3284725 /usr/bin/httpd -k start -DFOREGROUND
Apr 25 19:49:01 victoria systemd[1]: Started Apache Web Server.
# ----------------------------------------
# EXPORT CERTIFICATE FROM KEYSTORE:
# ---------------------------------
[victoria@victoria etc]$ keytool -exportcert -rfc -keystore solr-ssl.keystore.p12 -alias solr-ssl -file solr-ssl-cert
Enter keystore password: ## secret
Certificate stored in file <solr-ssl-cert>
[victoria@victoria etc]$ cat solr-ssl-cert
-----BEGIN CERTIFICATE-----
***
-----END CERTIFICATE-----
# ----------------------------------------
# ADD CERTIFICATE TO JAVA TRUST STORE (cacerts):
# ----------------------------------------------
## cacerts p/w generally defaults to: changeit
[victoria@victoria etc]$ sudo keytool -import -trustcacerts -cacerts -storepass *** -noprompt -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert
Certificate was added to keystore
[victoria@victoria etc]$
# ----------------------------------------
# VERIFY KEYSTORE ADDITION:
# -------------------------
[victoria@victoria etc]$ keytool -list -cacerts | grep solr-ssl
Enter keystore password: ***
solr-ssl, Apr 25, 2022, trustedCertEntry,
[victoria@victoria etc]$ keytool -list -keystore $JAVA_HOME/lib/security/cacerts | grep -i solr
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: ***
solr-ssl, Apr 25, 2022, trustedCertEntry,
[victoria@victoria etc]$
# ----------------------------------------
## Note: given the following [default installed] symlink,
[victoria@victoria solr-8.11.1]$ ls -l $JAVA_HOME/lib/security/cacerts
lrwxrwxrwx 1 root root 27 Apr 19 16:46 /usr/lib/jvm/java-18-openjdk//lib/security/cacerts -> /etc/ssl/certs/java/cacerts
[victoria@victoria solr-8.11.1]$
[victoria@victoria solr-8.11.1]$ keytool -list -keystore $JAVA_HOME/lib/security/cacerts | grep -i solr
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: ***
solr-ssl, Apr 24, 2022, trustedCertEntry,
[victoria@victoria solr-8.11.1]$
## ... importing the SSL certificate to $JAVA_HOME is not needed.
## old method [not needed / not done]:
sudo keytool -import -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.pem -alias solr-ssl -keystore $JAVA_HOME/lib/security/cacerts
## updated method [not needed / not done]:
sudo keytool -import -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert -keystore $JAVA_HOME/lib/security/cacerts
## Check [certificate present due to symlink, above]:
[victoria@victoria solr-8.11.1]$ sudo keytool -import -alias solr-ssl -file /mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl-cert -keystore $JAVA_HOME/lib/security/cacerts
[sudo] password for victoria:
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: ## ***
keytool error: java.lang.Exception: Certificate not imported, alias <solr-ssl> already exists
[victoria@victoria solr-8.11.1]$
# ============================================================================
# 2. INDEX DOCUMENTS TO SSL-HARDENED SOLR
# =======================================
[victoria@victoria scripts]$ date
2022-04-25 12:40:29 -0700 (PST)
[victoria@victoria scripts]$ apachectl configtest
Syntax OK
[victoria@victoria etc]$ pwd; ls -l
/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc
total 84K
-rw-r--r-- 1 victoria victoria 2.0K Dec 7 17:01 jetty-gzip.xml
-rw-r--r-- 1 victoria victoria 3.6K Dec 7 17:01 jetty-https8.xml
-rw-r--r-- 1 victoria victoria 3.7K Dec 7 17:01 jetty-https.xml
-rw-r--r-- 1 victoria victoria 2.7K Dec 7 17:01 jetty-http.xml
-rw-r--r-- 1 victoria victoria 1.9K Dec 7 17:01 jetty-requestlog.xml
-rw-r--r-- 1 victoria victoria 2.7K Apr 22 11:27 jetty-ssl.xml
-rw-r--r-- 1 victoria victoria 12K Apr 23 12:20 jetty.xml
-rw-r--r-- 1 victoria victoria 12K Dec 7 17:01 security.policy
-rw-r--r-- 1 victoria victoria 1.3K Dec 7 17:01 security.properties
-rw-r--r-- 1 victoria victoria 1.4K Apr 19 19:28 solr-ssl-cert
-rw-r--r-- 1 victoria victoria 2.8K Apr 19 19:27 solr-ssl.keystore.p12
-rw-r--r-- 1 victoria victoria 24K Dec 7 17:01 webdefault.xml
[victoria@victoria solr]$
# ----------------------------------------
# PATHS:
# ------
[victoria@victoria solr]$ echo $SOLR_JAVA_HOME/
/usr/lib/jvm/java-18-openjdk//
[victoria@victoria solr]$ echo $PATH | sed 's/:/\n/g' | grep solr
/mnt/Vancouver/apps/solr/solr-8.11.1/bin/
[victoria@victoria solr]$
# ----------------------------------------
# solr.in.sh :
# ------------
## Note: basic authentication allows access to SSL-protected Solr from the console / command-line.
SOLR_SSL_ENABLED=true
SOLR_SSL_KEY_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
SOLR_SSL_KEY_STORE_PASSWORD=secret
SOLR_SSL_KEY_STORE_TYPE=PKCS12
SOLR_SSL_TRUST_STORE=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
SOLR_SSL_TRUST_STORE_PASSWORD=secret
SOLR_SSL_TRUST_STORE_TYPE=PKCS12
SOLR_AUTH_TYPE="basic"
SOLR_AUTHENTICATION_OPTS="-Dbasicauth=pg-solr-admin:secret"
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false
SOLR_SSL_CLIENT_HOSTNAME_VERIFICATION=true
SOLR_SSL_CHECK_PEER_NAME=true
# ----------------------------------------
# security.json [note: p/w obfuscated (removed), below]:
# ------------------------------------------------------
{
"authentication":{
"blockUnknown":true,
"class":"solr.BasicAuthPlugin",
"credentials":{"pg-solr-admin":"***"},
"realm":"Persagen",
"forwardCredentials":false},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[
{
"name":"security-edit",
"role":"admin",
"index":1},
{
"name":"schema-read",
"role":["admin"],
"index":2},
{
"name":"security-read",
"role":["admin"],
"index":3},
{
"name":"all",
"role":["admin"],
"index":4}],
"user-role":{"pg-solr-admin":["admin"]},
"":{"v":0}}}
# ----------------------------------------
# JAVA VERSIONS:
# ---------------
[victoria@victoria solr]$ which java
/usr/bin/java
[victoria@victoria solr]$ whereis java
java: /usr/bin/java /usr/share/java /usr/lib/jvm/java-18-openjdk/bin/java /usr/share/man/man1/java.1.gz
[victoria@victoria solr]$
# ----------------------------------------
# SOLR INDEXING (old, for reference; note: http://...):
# -----------------------------------------------------
/usr/lib/jvm/java-8-openjdk/jre//bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.7.0/dist/solr-core-8.7.0.jar -Dauto=yes -Dc=core0 -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/programming/datasci/solr/test/d1.html /mnt/Vancouver/programming/datasci/solr/test/d2.html /mnt/Vancouver/programming/datasci/solr/test/d3.html /mnt/Vancouver/programming/datasci/solr/test/d4.html
/usr/lib/jvm/java-8-openjdk/jre//bin/java
-classpath /mnt/Vancouver/apps/solr/solr-8.7.0/dist/solr-core-8.7.0.jar
-Dauto=yes
-Dc=core0
-Ddata=files
org.apache.solr.util.SimplePostTool
/mnt/Vancouver/programming/datasci/solr/test/d1.html
/mnt/Vancouver/programming/datasci/solr/test/d2.html
/mnt/Vancouver/programming/datasci/solr/test/d3.html
/mnt/Vancouver/programming/datasci/solr/test/d4.html
# ----------------------------------------
# SOLR ADMIN UI (HINTS FOR PARAMETRIZATION):
# https://localhost:8983/solr/#/
# ------------------------------------------
-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Dbasicauth=pg-solr-admin:secret
-Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12
-Djavax.net.ssl.keyStoreType=PKCS12
[ ... snip ... ]
-Xms2048m
-Xmx2048m
-Xss256k
## NOTE: I want to run multiple Solr cores, and I am indexing HTML pages,
## metadata using bespoke { solrconfig.xml | schema.xml } files.
## For simplicity/testing, I temporarily moved to those *.xml files to:
## /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/
# ----------------------------------------------------------------------------
# DELETE OLD CONTENT (SOLR ADMIN UI; XML DOCUMENT TYPE):
# ------------------------------------------------------
<delete><query>*:*</query></delete>
# ----------------------------------------
# COMMAND-LINE POST TO SOLR:
# --------------------------
## Template:
/usr/lib/jvm/java-18-openjdk/bin/java \
-classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar \
-Dbasicauth=pg-solr-admin:secret \
-Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ \
-Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.keyStoreType=PKCS12 \
-Djavax.net.ssl.keyStorePassword=secret \
-Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.trustStoreType=PKCS12 \
-Djavax.net.ssl.trustStorePassword=secret \
-Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server \
-Djetty.port=8983 \
-Dsolr.jetty.https.port=8983 \
-Dsolr.jetty.keystore.type=PKCS12 \
-Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.jetty.ssl.needClientAuth=false \
-Dsolr.jetty.ssl.wantClientAuth=false \
-Dsolr.jetty.truststore.type=PKCS12 \
-Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr \
-Dauto=yes \
-Dc=pg-solr \
-Durl=https://localhost:8983/solr/pg-solr/update \
-Ddata=files \
org.apache.solr.util.SimplePostTool \
/mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
# ----------------------------------------
# INDEX DOCUMENT TO "pg-solr" CORE [SUCCESS! 😀]
# ----------------------------------------------
[victoria@victoria solr]$ solr status
Found 1 Solr nodes:
Solr process 272444 running on port 8983
INFO - 2022-04-26 10:26:55.684; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
INFO - 2022-04-26 10:26:55.686; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
{
"solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
"version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
"startTime":"2022-04-26T16:27:45.310Z",
"uptime":"0 days, 0 hours, 59 minutes, 10 seconds",
"memory":"251.5 MB (%12.3) of 2 GB"}
[victoria@victoria solr]$
## POST HTML DOCUMENT:
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java \
-classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar \
-Dbasicauth=pg-solr-admin:secret \
-Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ \
-Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.keyStoreType=PKCS12 \
-Djavax.net.ssl.keyStorePassword=secret \
-Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Djavax.net.ssl.trustStoreType=PKCS12 \
-Djavax.net.ssl.trustStorePassword=secret \
-Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server \
-Djetty.port=8983 \
-Dsolr.jetty.https.port=8983 \
-Dsolr.jetty.keystore.type=PKCS12 \
-Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.jetty.ssl.needClientAuth=false \
-Dsolr.jetty.ssl.wantClientAuth=false \
-Dsolr.jetty.truststore.type=PKCS12 \
-Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 \
-Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr \
-Dauto=yes \
-Dc=pg-solr \
-Durl=https://localhost:8983/solr/pg-solr/update \
-Ddata=files \
org.apache.solr.util.SimplePostTool \
/mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file jeff_bezos.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
Time spent: 0:00:01.214
# ----------------------------------------
# REPEAT (TEST: DID NOT DUPLICATE EXISTING INDEXED ENTRY 👍):
# -----------------------------------------------------------
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/jeff_bezos.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file jeff_bezos.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
Time spent: 0:00:02.336
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
# CREATE NEW SOLR CORE:
# ---------------------
[victoria@victoria solr]$ solr create -c pg-solr-demo
WARNING: Using _default configset with data driven schema functionality. NOT RECOMMENDED for production use.
To turn off: bin/solr config -c pg-solr-demo -p 8983 -action set-user-property -property update.autoCreateFields -value false
INFO - 2022-04-26 11:22:02.802; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
INFO - 2022-04-26 11:22:02.804; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
Created new core 'pg-solr-demo'
[victoria@victoria solr]$ solr restart
Sending stop command to Solr running on port 8983 ... waiting up to 45 seconds to allow Jetty process 272444 to stop gracefully.
Waiting up to 45 seconds to see Solr running on port 8983 [|]
Started Solr server on port 8983 (pid=446422). Happy searching!
[victoria@victoria solr]$ solr status
Found 1 Solr nodes:
Solr process 446422 running on port 8983
INFO - 2022-04-26 11:22:25.221; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
INFO - 2022-04-26 11:22:25.223; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
{
"solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
"version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
"startTime":"2022-04-26T18:22:12.715Z",
"uptime":"0 days, 0 hours, 0 minutes, 13 seconds",
"memory":"54.2 MB (%2.6) of 2 GB"}
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
# INDEX NEW DOCUMENT TO ORIGINAL CORE [pg-solr]:
# ----------------------------------------------
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/greg_abbott.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file greg_abbott.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr/update...
Time spent: 0:00:02.287
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
# INDEX NEW DOCUMENT TO NEW CORE [pg-solr-demo]:
# ----------------------------------------------
[victoria@victoria solr]$ /usr/lib/jvm/java-18-openjdk/bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dbasicauth=pg-solr-admin:secret -Dsolr.default.confdir=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/configsets/_default/conf/ -Djavax.net.ssl.keyStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.keyStoreType=PKCS12 -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.trustStore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=secret -Djetty.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server -Djetty.port=8983 -Dsolr.jetty.https.port=8983 -Dsolr.jetty.keystore.type=PKCS12 -Dsolr.jetty.keystore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Dsolr.jetty.truststore.type=PKCS12 -Dsolr.jetty.truststore=/mnt/Vancouver/apps/solr/solr-8.11.1/server/etc/solr-ssl.keystore.p12 -Dsolr.solr.home=/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr -Dauto=yes -Dc=pg-solr -Durl=https://localhost:8983/solr/pg-solr-demo/update -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/domains/persagen/05/docs-test_copies/neofascism.html
SimplePostTool version 5.0.0
Basic Authentication enabled, user=pg-solr-admin
Posting files to [base] url https://localhost:8983/solr/pg-solr-demo/update...
Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
POSTing file neofascism.html (text/html) to [base]/extract
1 files indexed.
COMMITting Solr index changes to https://localhost:8983/solr/pg-solr-demo/update...
Time spent: 0:00:01.715
[victoria@victoria solr]$
# ----------------------------------------------------------------------------
## Q.E.D 😅
# ============================================================================
Re: Cannot post to SSL-secured core from command line
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Victoria,
On 4/26/22 00:15, Victoria Stuart (VictoriasJourney.com) wrote:
>
> With a standalone Solr 8.11.1 instance with SSL enabled on an Arch Linux system,
>
> [victoria@victoria solr-8.11.1]$ solr status
>
> Found 1 Solr nodes:
>
> Solr process 3380449 running on port 8983
> INFO - 2022-04-25 20:55:03.283; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
> INFO - 2022-04-25 20:55:03.286; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
> {
> "solr_home":"/mnt/Vancouver/apps/solr/solr-8.11.1/server/solr",
> "version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
> "startTime":"2022-04-26T03:52:16.715Z",
> "uptime":"0 days, 0 hours, 2 minutes, 47 seconds",
> "memory":"89.3 MB (%4.4) of 2 GB"}
>
> [victoria@victoria solr-8.11.1]$
>
> ... how can I post documents on the command line?
>
> Do I need to provide my Solr admin username:password ( -u user;pass)?
>
> Here I show the output without a username:password, but I got the identical
> output using it.
>
> [victoria@victoria solr-8.11.1]$ post -c pg-solr /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/virginia_lamp_thomas.html
>
> /usr/lib/jvm/java-18-openjdk//bin/java -classpath /mnt/Vancouver/apps/solr/solr-8.11.1/dist/solr-core-8.11.1.jar -Dauto=yes -Dc=pg-solr -Ddata=files org.apache.solr.util.SimplePostTool /mnt/Vancouver/apps/solr/solr-8.11.1/server/solr/virginia_lamp_thomas.html
> SimplePostTool version 5.0.0
> Posting files to [base] url http://localhost:8983/solr/pg-solr/update...
> Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
> POSTing file virginia_lamp_thomas.html (text/html) to [base]/extract
> SimplePostTool: FATAL: IOException while posting data: java.io.IOException: Error writing request body to server
You are using client TLS certificates, right?
It looks like the tool itself doesn't have direct CLI support for it,
you have to add your own system properties from the CLI:
https://solr.apache.org/guide/8_8/enabling-ssl.html#index-documents-using-post-jar
e.g.
$ java -Djavax.net.ssl.keyStorePassword=secret
-Djavax.net.ssl.keyStore=../../server/etc/solr-ssl.keystore.p12
-Djavax.net.ssl.trustStore=../../server/etc/solr-ssl.keystore.p12
-Djavax.net.ssl.trustStorePassword=secret
-Durl=https://localhost:8984/solr/mycollection/update -jar post.jar *.xml
Note that the "key store" here must point to your *client keystore*
containing your client key + cert and the trust store must point to a
file containing your server's certificate (and, optionally, key).
You can also use curl which may be more convenient than building-up the
whole Java command-line like that.
If you aren't using client certs, you will probably still need to
specify the trust store on the CLI unless you have a server cert signed
by a real CA. (Most Solrs I've seen just use self-signed certs becasue
there really isn't a reason to have the signed by a real CA.)
-chris