You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ja...@apache.org on 2018/09/04 12:23:54 UTC
[1/2] lucene-solr:branch_7x: SOLR-11690: Improve documentation about
DIH password encryption
Repository: lucene-solr
Updated Branches:
refs/heads/branch_7x cc573d7bb -> 531fbd173
SOLR-11690: Improve documentation about DIH password encryption
(cherry picked from commit d997e8b)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/531fbd17
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/531fbd17
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/531fbd17
Branch: refs/heads/branch_7x
Commit: 531fbd17331b3ca73c4a8d9d6e3bd399e81acda2
Parents: 4d588fc
Author: Jan Høydahl <ja...@apache.org>
Authored: Tue Sep 4 14:22:32 2018 +0200
Committer: Jan Høydahl <ja...@apache.org>
Committed: Tue Sep 4 14:23:47 2018 +0200
----------------------------------------------------------------------
...store-data-with-the-data-import-handler.adoc | 28 +++++++++++++-------
1 file changed, 18 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/531fbd17/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc b/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc
index 7cf50ee..cdfee7d 100644
--- a/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc
+++ b/solr/solr-ref-guide/src/uploading-structured-data-store-data-with-the-data-import-handler.adoc
@@ -141,21 +141,29 @@ http://localhost:8983/solr/dih/dataimport?command=full-import&jdbcurl=jdbc:hsqld
==== Encrypting a Database Password
-The database password can be encrypted if necessary to avoid plaintext passwords being exposed in unsecured files. To do this, follow these steps:
-
-. In a terminal window, run the command `openssl enc -aes-128-cbc -a -salt -in pwd.txt`.
-..This assumes the password is in a file named `pwd.txt`. If you don't have the password in this file yet, you can do `echo "mypassword" > pwd.txt`.
-.. The openssl session will ask for a password to use for the decryption. You will use this file with a `encryptKeyFile` parameter in `data-config.xml`.
-.. The output of the process will be a long string such as `U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o=`. This will be the password you put in your `data-config.xml` file.
-. Save the password you used as the decryption password in the previous step to a file, and determine the location of the file on the Solr server. You could use a command such as `echo myencrypfilepwd > /location/of/encryptionkey`. Replace "myencrypfilepwd" with the password you used while generating the key.
-. If the file is not yet on the Solr server, move it there. Also make sure the encryption key file permissions do not allow it to be read by unauthorized users. The `chmod 0600` command should set the permissions sufficiently.
+The database password can be encrypted if necessary to avoid plaintext passwords being exposed in unsecured files. To do this, we will replace the password in `data-config.xml` with an encrypted password. We will use the `openssl` tool for the encryption, and the encryption key will be stored in a file which is only readable to the `solr` process. Please follow these steps:
+
+. Create a strong encryption password and store it in a file. Then make sure it is readable only for the `solr` user. Example commands:
+
+ echo -n "a-secret" > /var/solr/data/dih-encryptionkey
+ chown solr:solr /var/solr/data/dih-encryptionkey
+ chmod 600 /var/solr/data/dih-encryptionkey
+
+. Encrypt the JDBC database password using `openssl` as follows:
+
+ echo -n "my-jdbc-password" | openssl enc -aes-128-cbc -a -salt -md md5 -pass file:/var/solr/data/dih-encryptionkey
+
+.. The output of the command will be a long string such as `U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o=`. You will use this as `password` in your `data-config.xml` file.
. In your `data-config.xml`, you'll add the `password` and `encryptKeyFile` parameters to the `<datasource>` configuration, as in this example:
+
[source,xml]
<dataSource driver="org.hsqldb.jdbcDriver"
url="jdbc:hsqldb:./example-DIH/hsqldb/ex"
- user="sa" password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
- encryptKeyFile="/location/of/encryptionkey />
+ user="sa"
+ password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
+ encryptKeyFile="/var/solr/data/dih-encryptionkey" />
+
+NOTE: Note that we use the `-n` argument to `echo` to avoid including a newline character at the end of the password. If you use another method to generate the encrypted password, make sure to avoid newlines as well.
== DataImportHandler Commands
[2/2] lucene-solr:branch_7x: LUCENE-5143: rm or formalize dealing
with "general" KEYS files in our dist dir
Posted by ja...@apache.org.
LUCENE-5143: rm or formalize dealing with "general" KEYS files in our dist dir
(cherry picked from commit 982ee39)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/4d588fcd
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/4d588fcd
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/4d588fcd
Branch: refs/heads/branch_7x
Commit: 4d588fcd8151be308c3ecec134ddc2a6afb0531b
Parents: cc573d7
Author: Jan Høydahl <ja...@apache.org>
Authored: Tue Sep 4 13:24:24 2018 +0200
Committer: Jan Høydahl <ja...@apache.org>
Committed: Tue Sep 4 14:23:47 2018 +0200
----------------------------------------------------------------------
dev-tools/scripts/buildAndPushRelease.py | 54 ++++++++++++++++++++++-----
1 file changed, 45 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/4d588fcd/dev-tools/scripts/buildAndPushRelease.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/buildAndPushRelease.py b/dev-tools/scripts/buildAndPushRelease.py
index 417bd3d..3694f1a 100644
--- a/dev-tools/scripts/buildAndPushRelease.py
+++ b/dev-tools/scripts/buildAndPushRelease.py
@@ -64,12 +64,12 @@ def runAndSendGPGPassword(command, password):
print(msg)
raise RuntimeError(msg)
-def load(urlString):
+def load(urlString, encoding="utf-8"):
try:
- content = urllib.request.urlopen(urlString).read().decode('utf-8')
+ content = urllib.request.urlopen(urlString).read().decode(encoding)
except Exception as e:
print('Retrying download of url %s after exception: %s' % (urlString, e))
- content = urllib.request.urlopen(urlString).read().decode('utf-8')
+ content = urllib.request.urlopen(urlString).read().decode(encoding)
return content
def getGitRev():
@@ -218,12 +218,6 @@ def pushLocal(version, root, rev, rcNum, localDir):
run('tar xjf "%s/solr/package/solr.tar.bz2"' % root)
os.remove('%s/solr/package/solr.tar.bz2' % root)
- print(' KEYS')
- run('wget http://home.apache.org/keys/group/lucene.asc')
- os.rename('lucene.asc', 'KEYS')
- run('chmod a+r-w KEYS')
- run('cp KEYS ../lucene')
-
print(' chmod...')
os.chdir('..')
run('chmod -R a+rX-w .')
@@ -245,6 +239,8 @@ def parse_config():
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('--no-prepare', dest='prepare', default=True, action='store_false',
help='Use the already built release in the provided checkout')
+ parser.add_argument('--local-keys', metavar='PATH',
+ help='Uses local KEYS file to validate presence of RM\'s gpg key')
parser.add_argument('--push-local', metavar='PATH',
help='Push the release to the local path')
parser.add_argument('--sign', metavar='KEYID',
@@ -263,6 +259,8 @@ def parse_config():
parser.error('Release Candidate number must be a positive integer')
if not os.path.isdir(config.root):
parser.error('Root path "%s" is not a directory' % config.root)
+ if config.local_keys is not None and not os.path.exists(config.local_keys):
+ parser.error('Local KEYS file "%s" not found' % config.local_keys)
cwd = os.getcwd()
os.chdir(config.root)
config.root = os.getcwd() # Absolutize root dir
@@ -298,11 +296,49 @@ def check_ant():
return
raise RuntimeError('Unsupported ant version (must be 1.8 - 1.10): "%s"' % antVersion)
+def check_key_in_keys(gpgKeyID, local_keys):
+ if gpgKeyID is not None:
+ print(' Verify your gpg key is in the main KEYS file')
+ if local_keys is not None:
+ print(" Using local KEYS file %s" % local_keys)
+ keysFileText = open(local_keys, encoding='iso-8859-1').read()
+ keysFileLocation = local_keys
+ else:
+ keysFileURL = "https://archive.apache.org/dist/lucene/KEYS"
+ keysFileLocation = keysFileURL
+ print(" Using online KEYS file %s" % keysFileURL)
+ keysFileText = load(keysFileURL, encoding='iso-8859-1')
+ if len(gpgKeyID) > 2 and gpgKeyID[0:2] == '0x':
+ gpgKeyID = gpgKeyID[2:]
+ if len(gpgKeyID) > 40:
+ gpgKeyID = gpgKeyID.replace(" ", "")
+ if len(gpgKeyID) == 8:
+ re_to_match = r"^pub\s+\d+[DR]/%s " % gpgKeyID
+ elif len(gpgKeyID) == 40:
+ gpgKeyID40Char = "%s %s %s %s %s %s %s %s %s %s" % \
+ (gpgKeyID[0:4], gpgKeyID[4:8], gpgKeyID[8:12], gpgKeyID[12:16], gpgKeyID[16:20],
+ gpgKeyID[20:24], gpgKeyID[24:28], gpgKeyID[28:32], gpgKeyID[32:36], gpgKeyID[36:])
+ print("Generated id string %s" % gpgKeyID40Char)
+ re_to_match = r"^\s+Key fingerprint = %s$" % gpgKeyID40Char
+ else:
+ print('Invalid gpg key id format. Must be 8 byte short ID or 40 byte fingerprint, with or without 0x prefix.')
+ exit(2)
+ if re.search(re_to_match, keysFileText, re.MULTILINE):
+ print(' Found key %s in KEYS file at %s' % (gpgKeyID, keysFileLocation))
+ else:
+ print(' ERROR: Did not find your key %s in KEYS file at %s. Please add it and try again.' % (gpgKeyID, keysFileLocation))
+ if local_keys is not None:
+ print(' You are using a local KEYS file. Make sure it is up to date or validate against the online version')
+ exit(2)
+
+
def main():
check_cmdline_tools()
c = parse_config()
+ check_key_in_keys(c.key_id, c.local_keys)
+
if c.prepare:
rev = prepare(c.root, c.version, c.key_id, c.key_password)
else: