You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by alopresto <gi...@git.apache.org> on 2016/02/02 02:41:09 UTC
[GitHub] nifi pull request: NIFI-1257 and 1259
GitHub user alopresto opened a pull request:
https://github.com/apache/nifi/pull/201
NIFI-1257 and 1259
NIFI-1257
NIFI-1259
Added the mechanics for additional strong `Key Derivation Functions` (KDF) including `PBKDF2`, `Bcrypt`, and `Scrypt` as well as `keyed ciphers` (raw key; no KDF). Integrated with `EncryptContent` processor and added new property and property descriptors. Dependent properties not available until NIFI-1121 is implemented.
Explanation of KDFs available here: [Apache NiFi Wiki](https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/alopresto/nifi NIFI-1257-cherrypick
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/201.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #201
----
commit ab566da624914d553188495a906a523ce7cbe209
Author: Andy LoPresto <al...@gmail.com>
Date: 2015-12-22T18:12:02Z
NIFI-1257:
Added first skeleton of KeyDeriver functionality but re-evaluating to handle OpenSSL and NiFi legacy PBE consistently with modern KDFs.
commit aaef4ec6e5551392fdf441fe2bca5f6a924206ea
Author: Andy LoPresto <al...@gmail.com>
Date: 2015-12-22T19:02:27Z
NIFI-1257 Skeleton of key deriver mechanics and moving existing classes into crypto package.
Added variable substitution with defaults to logback-test.xml. (+11 squashed commits)
Squashed commits:
[5aa2ae2] NIFI-1257:
Minor formatting changes in logback-test.xml.
[af12eb5] NIFI-1257:
Added CipherFactory.
[3e04f5b] NIFI-1257:
Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider.
[bfdc1d7] NIFI-1257:
Added PBKDF2 cipher provider implementation and Java & Groovy unit test skeletons.
[20adeef] NIFI-1257:
Added KeyedCipherProvider interface.
[302a864] NIFI-1257:
Moved OpenPGP encryptor unit tests to proper package.
[ce5c231] NIFI-1257:
Added test logback file with variable substitution expressions.
[f2475eb] NIFI-1257:
Added Groovy unit test support to allow faster development cycle.
[4004cfd] NIFI-1257:
Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test.
[332b9dc] NIFI-1257:
Moved crypto classes into new package.
Began refactor from KeyDeriver to CipherProvider.
[5277efc] NIFI-1257:
Added variables with default value to logback-test.xml.
commit 07c1e1219153ee3ab5a030f68f0647ad41bdd3b9
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-13T20:25:34Z
NIFI-1257 Committed state before refactor to inherit from shared CipherProvider interface. (+5 squashed commits)
Squashed commits:
[634802c] NIFI-1257 Added new keyed encryption methods and added boolean field for compatibility with new KDFs.
[b74d23c] NIFI-1257 Added parseKeyLengthFromAlgorithm utility method.
Added unit tests.
[f3cd3b6] NIFI-1257 Moved cipher algorithm parsing utility method to common util class.
[b0e785e] NIFI-1257 Applied contrib-check fixes for line length.
[94a6ecf] NIFI-1257 Added Apache License to various tests.
commit 106de28eeda3838413226185dfe52e0c83bbf04a
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-14T02:00:34Z
NIFI-1257 Implemented PBKDF2 cipher provider.
Added default constructor with strong choices for PBKDF2 cipher provider. (+12 squashed commits)
Squashed commits:
[19f94a0] NIFI-1257 Implemented NiFiLegacyCipherProvider and added unit tests.
[efcb5f6] NIFI-1257 Added convenience methods in OpenSSLPKCS5 provider that ignore desired key lengths (all OpenSSL algorithms specify key length in algorithm).
[0381a06] NIFI-1257 Moved BC provider loading in test to single static invocation (decreases test time by ~40%).
[b5a8682] NIFI-1257 Added ASF license to openssl_pbkdf2.rb.
[87cfca0] NIFI-1257 Test cleanup.
[cb3598b] NIFI-1257 Added key length parameter to PBKDF2 cipher provider.
[aa50006] NIFI-1257 Added PRF resolution to PBKDF2 cipher provider.
[9dfb6b0] NIFI-1257 Changed PBKDF2 and OpenSSLPKCS5 implementations to accept EncryptionMethod instead of individual algorithm and provider combinations.
[d6156bd] NIFI-1257 Added test resource to generate PBKDF2-derived key and encrypt content using OpenSSL via Ruby.
[6cee174] NIFI-1257 Removed duplicated test.
[5ff87a4] NIFI-1257 Temporarily commented out other implementations while interface changes.
[2f1ac0c] NIFI-1257 Added RandomIVPBECipherProvider to allow for non-deterministic IVs.
commit 3a835c4c9a8390a41090b7a66e905a3985b39977
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-17T06:43:25Z
NIFI-1257 Added Bcrypt implementation.
Added unit tests.
Added jBcrypt dependency to pom because BC Bcrypt is OpenBSD-based, not compatible with standard Bcrypt as defined by the Niels Provos paper.
Added Ruby script in resources for external compatibility testing.
commit a7c9a4a63facb82d5037c47332230c8b9200ca40
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-19T03:52:18Z
NIFI-1257 Added Javadoc, unit tests, and utility methods.
Updated PBKDF2 IV check to invalid length, rather than just non-zero check. (+15 squashed commits)
Squashed commits:
[0c66002] NIFI-1257 Added isKeyedCipher boolean method for EncryptionMethod.
[5d64dfa] NIFI-1257 Added TODO for invalid IV length check.
[5c73364] NIFI-1257 Added test for empty PRF in PBKDF2 constructor.
Added key length check to PBKDF2 cipher provider.
[57ade68] NIFI-1257 Ignored tests for weak iteration count/work factor but added warning message in constructors.
Changed default PRF to SHA-512.
Added salt and key length check to PBKDF2 cipher provider.
[362828f] NIFI-1257 Added pass-through of IllegalArgumentExceptions in cipher initialization.
[9f019ed] NIFI-1257 OpenSSL cipher provider ignores requested key length.
[6db634b] NIFI-1257 Added check for salt length and encryption method presence.
[e6426a9] NIFI-1257 Added utility method for algorithm-specific key length check.
Added unit tests.
[3503b35] NIFI-1257 Added unit test validating salt format for Bcrypt.
[9ffd358] NIFI-1257 Implemented key length check in Bcrypt cipher provider.
Added unit tests.
[5dc8801] NIFI-1257 Improved Javadoc on CipherUtility.
[a3eb290] NIFI-1257 Added utility method to check key length validity for cipher families.
Added unit tests.
[51ce74f] NIFI-1257 Added unit tests to enforce strong default values for Bcrypt work factor and PBKDF2 iteration counts.
[b8dfd4e] NIFI-1257 Corrected license language for jBcrypt in nifi-standard-nar and nifi-assembly LICENSE files.
[8a85b33] NIFI-1257 Added license info for jBcrypt into nifi-standard-nar and nifi-assembly LICENSE files.
commit 28243e9663e53fc8232bff21a65f1bd7654e32e0
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-21T03:48:45Z
NIFI-1257 Reduced code duplication by invoking KeyedCipherProvider from BcryptCipherProvider after key derivation. (+4 squashed commits)
Squashed commits:
[1c2f06f] NIFI-1257 Fixed bug in Base64 section of Bcrypt salt regular expression.
[632104a] NIFI-1257 Added unit test for generateSalt.
[8cc990f] NIFI-1257 Reduced code duplication by invoking KeyedCipherProvider from PBKDF2CipherProvider after key derivation.
Added unit test for invalid IV.
[75c4690] NIFI-1257 Defined interface for KeyedCipherProvider.
Implemented AES implementation for KeyedCipherProvider.
Added unit tests.
Added Ruby script to test/resources for external compatibility check.
commit 3f0b3ca30423000c50075414d4ca02bb21bc69cd
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-22T03:35:43Z
NIFI-1257 Implemented Scrypt cipher provider.
Refactored Scrypt internals and added unit tests.
Added unit test for Scrypt check with correct and incorrect passwords.
Changed check to remove hard-coded derived key length. (+3 squashed commits)
Squashed commits:
[c1d7637] NIFI-1257 Added unit test for Scrypt check with invalid passwords and hashes.
[f89b2fb] NIFI-1257 Implemented minimum safe parameters calculator and unit test.
Updated salt format regex.
[f553bce] NIFI-1257 Moved memory calculation helper to Scrypt. (+2 squashed commits)
Squashed commits:
[13c1b52] NIFI-1257 Cleaned up unit tests and javadoc.
Resolved failing PBKDF2 tests and copied multiple run average logic to Bcrypt tests. (+6 squashed commits)
Squashed commits:
[b16a5b3] NIFI-1257 Updated default iteration count for PBKDF2 cipher provider.
[b083b66] NIFI-1257 Refactored parameter encoding to separate method.
[5369fbb] NIFI-1257 Updated Javadoc.
[90feb2a] NIFI-1257 Added check for all-zero IV (common security vulnerability) in AES cipher provider.
Added unit test.
[7cf6c78] NIFI-1257 Updated Javadoc to be consistent across implementations.
[c0424d6] NIFI-1257 Deprecated no-salt cipher generator for Bcrypt to be consistent with Scrypt implementation.
Updated unit tests.
[06bed9a] NIFI-1257 Implemented Scrypt cipher provider.
Added unit tests for empty salt. (+7 squashed commits)
Squashed commits:
[130b477] NIFI-1257 Added salt translator from mcrypt format to Java format.
Added unit tests for salt formatting and validation.
[c43ba44] NIFI-1257 Implemented ScryptCipherProvider and added unit test skeleton with internal consistency and external compatibility tests.
[9db678f] NIFI-1257 Updated Bcrypt unit test to accept new interface method contract.
[897ad03] NIFI-1257 Added sanity test for Scrypt primitive key derivation compatibility with external Ruby library.
[4adc3fe] NIFI-1257 Exposed Scrypt parseSalt method as public to allow ScryptCipherProvider to accept formatted salt externally.
[391cd8b] NIFI-1257 Updated scrypt Ruby script to be consistent with bcrypt and pbkdf2 scripts in key and IV data handling.
[d167917] NIFI-1257 Exposed desired key length as parameter on Scrypt underlying implementation. (+2 squashed commits)
Squashed commits:
[7f1feb5] NIFI-1257 Standardized default salt generation to byte[] and added method contracts in top-level PBECipherProvider interface.
[673f35f] NIFI-1257 Scrypt implementation.
Added default salt length constant and accessor in Scrypt.
Changed reference to UTF-8 to be enum instead of freeform requiring runtime resolution and redundant exception handling. (+11 squashed commits)
Squashed commits:
[b48ac88] NIFI-1257 Removed salt validation check in Scrypt (spec allows for empty salt) and replaced with logger warnings. Moved unit test to ScryptCipherProvider unit test as enforcement will be moved higher up.
[611abf6] NIFI-1257 Made convention & style changes in scrypt code.
[9005573] NIFI-1257 Added salt validation and unit test for Scrypt.
[ff12568] NIFI-1257 Added validation check for password and relevant unit test.
[6b42e8f] NIFI-1257 Added validation check for N, r, p in Scrypt and relevant unit tests.
[8ab7e45] NIFI-1257 Added ruby script to test external compatibility for Scrypt implementation.
[b720b30] NIFI-1257 Combined ScryptUtil (application-facing interface) into Scrypt class to reduce unnecessary complexity.
Updated unit test references.
[7739349] NIFI-1257 Moved high-memory test vector into separate unit test with Assume statement limiting execution depending on heap size to allow faster execution.
[badd2ab] NIFI-1257 Enabled high-memory test vector for Scrypt test.
[765cd38] NIFI-1257 Added surefire block to groovy unit test profile to enforce 3072 MB heap for Scrypt test.
[637f80b] NIFI-1257 Added local Java implementation of Scrypt KDF (and underlying PBKDF2 KDF) from Will Glozer.
Added unit tests (including test vectors from paper).
commit f875ecac6c8bf2704a2d662587b7b635e2ee7e39
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-29T22:52:12Z
NIFI-1257 Cleaned up interfaces and refactored shared code/unit and regression tests.
Refactored shared stream handling code. (+1 squashed commit)
Squashed commits:
[4c6de17] NIFI-1257 Added logic to Encryptor to read and write IV to/from cipher stream.
Added unit tests to cover all PBE KDFs. (+1 squashed commit)
Squashed commits:
[f333fa0] NIFI-1257 Added methods to read and write IV to/from cipher stream. (+12 squashed commits)
Squashed commits:
[4ece756] NIFI-1257 Resolved all tests in nifi-standard-processors module.
[8163f07] NIFI-1257 Refactored implementation-specific logic out of PasswordBasedEncryptor to support all KDFs.
Regression tests for legacy still pass.
[a52da26] NIFI-1257 Added salt read/write logic to NifiLegacy and OpenSSL cipher providers.
Added common salt read/write logic to RandomIVPBECipherProvider.
Changed RandomIVPBECipherProvider from interface to abstract class.
Updated strong KDF implementations.
[1407afe] NIFI-1257 Added common cipher processing code to CipherUtility.
[442495b] NIFI-1257 Added regression tests for PasswordBasedEncryptor with legacy OpenSSL compatibility before integrating cipher provider logic.
[d0410c0] NIFI-1257 Added regression test for PasswordBasedEncryptor before integrating cipher provider logic.
[e586933] NIFI-1257 Removed legacy unit test (superseded by Groovy unit test).
[170314a] NIFI-1257 Removed unnecessary EncryptionMethod enumerations from unit test.
[196d560] NIFI-1257 Added ASF 2.0 license to CipherProfiderFactory unit test.
[36e95ba] NIFI-1257 Removed unnecessary KeyDeriver interface.
[9c4eb07] NIFI-1257 Added default (NONE) KDF enum value.
Renamed CipherFactory to CipherProviderFactory.
Added unit test for registered KDF resolution from factory.
[f006985] NIFI-1257 Added CipherProvider marker interface which PBE and keyed cipher providers share.
commit d45dc69b3981ed6194a90aedbf603122d5c47c18
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-30T06:03:25Z
NIFI-1257 Added IV read/write to KeyedCipherProvider and changed from interface to abstract class. (+3 squashed commits)
Squashed commits:
[0ee5175] NIFI-1257 Moved null checks for encryption method and password to constructor to de-duplicate and catch earlier.
[6ed4142] NIFI-1257 Moved stream handling code to CipherUtility to allow reuse in KeyedCipherProvider.
[a78b5db] NIFI-1257 Added null checks for encryption method and password.
commit 48b3463319a8fc620a9229a28253e4c99f4afc8e
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-01-30T07:27:46Z
NIFI-1257 Added processor logic to invoke keyed cipher.
Added unit tests. (+5 squashed commits)
Squashed commits:
[28b2224] NIFI-1257 Fixed contrib-check issues.
[143b88f] NIFI-1257 Added helper method on KDF enum to indicate strength of KDF.
Added EncryptContent processor property for raw hex key (always visible until NIFI-1121).
Added validations for KDF (keyed and PBE) and hex key.
Added unit tests for EncryptContent processor validations.
[3a8367b] NIFI-1257 Added utility method to return list of valid key lengths for algorithm.
Added unit tests.
[04d6d16] NIFI-1257 Added description to allowable values for KDF and encryption method in EncryptContent processor.
[98595d8] NIFI-1257 Added KeyedEncryptor and unit tests.
commit 3c410d8f33b3ceaf5dfd79db355cf087edec80c8
Author: Andy LoPresto <al...@gmail.com>
Date: 2016-02-02T01:25:28Z
NIFI-1257 Fixed casing of Scrypt.java.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51633399
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/CipherUtility.java ---
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import javax.crypto.Cipher;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class CipherUtility {
+
+ public static final int BUFFER_SIZE = 65536;
+
+ /**
+ * Returns the cipher algorithm from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> AES
+ *
+ * @param algorithm the full algorithm name
+ * @return the generic cipher name or the full algorithm if one cannot be extracted
+ */
+ public static String parseCipherFromAlgorithm(final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return algorithm;
+ }
+ String formattedAlgorithm = algorithm.toUpperCase();
+
+ // This is not optimal but the algorithms do not have a standard format
+ final String AES = "AES";
+ final String TDES = "TRIPLEDES";
+ final String TDES_ALTERNATE = "DESEDE";
+ final String DES = "DES";
+ final String RC4 = "RC4";
+ final String RC2 = "RC2";
+ final String TWOFISH = "TWOFISH";
+ final List<String> SYMMETRIC_CIPHERS = Arrays.asList(AES, TDES, TDES_ALTERNATE, DES, RC4, RC2, TWOFISH);
+
+ // The algorithms contain "TRIPLEDES" but the cipher name is "DESede"
+ final String ACTUAL_TDES_CIPHER = "DESede";
+
+ for (String cipher : SYMMETRIC_CIPHERS) {
+ if (formattedAlgorithm.contains(cipher)) {
+ if (cipher.equals(TDES) || cipher.equals(TDES_ALTERNATE)) {
+ return ACTUAL_TDES_CIPHER;
+ } else {
+ return cipher;
+ }
+ }
+ }
+
+ return algorithm;
+ }
+
+ /**
+ * Returns the cipher key length from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> 128
+ *
+ * @param algorithm the full algorithm name
+ * @return the key length or -1 if one cannot be extracted
+ */
+ public static int parseKeyLengthFromAlgorithm(final String algorithm) {
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ return keyLength;
+ } else {
+ // Key length not explicitly named in algorithm
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ return getDefaultKeyLengthForCipher(cipher);
+ }
+ }
+
+ private static int parseActualKeyLengthFromAlgorithm(final String algorithm) {
+ Pattern pattern = Pattern.compile("([\\d]+)BIT");
+ Matcher matcher = pattern.matcher(algorithm);
+ if (matcher.find()) {
+ return Integer.parseInt(matcher.group(1));
+ } else {
+ return -1;
+ }
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided cipher family. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * Does not reflect if the key length is correct for a specific combination of cipher and PBE-derived key length.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}. However, this method will return {@code true} for both because it only gets the cipher
+ * family, {@code AES}.
+ * <p/>
+ * 64, AES -> false
+ * [128, 192, 256], AES -> true
+ *
+ * @param keyLength the key length in bits
+ * @param cipher the cipher family
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLength(int keyLength, final String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(cipher).contains(keyLength);
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided algorithm. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}.
+ * <p/>
+ * 64, AES/CBC/PKCS7Padding -> false
+ * [128, 192, 256], AES/CBC/PKCS7Padding -> true
+ * <p/>
+ * 128, PBEWITHMD5AND128BITAES-CBC-OPENSSL -> true
+ * [192, 256], PBEWITHMD5AND128BITAES-CBC-OPENSSL -> false
+ *
+ * @param keyLength the key length in bits
+ * @param algorithm the specific algorithm
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLengthForAlgorithm(int keyLength, final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(algorithm).contains(keyLength);
+ }
+
+ public static List<Integer> getValidKeyLengthsForAlgorithm(String algorithm) {
+ List<Integer> validKeyLengths = new ArrayList<>();
+ if (StringUtils.isEmpty(algorithm)) {
+ return validKeyLengths;
+ }
+
+ // Some algorithms specify a single key size
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ validKeyLengths.add(keyLength);
+ return validKeyLengths;
+ }
+
+ // The algorithm does not specify a key size
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ switch (cipher.toUpperCase()) {
+ case "DESEDE":
+ // 3DES keys have the cryptographic strength of 7/8 because of parity bits, but are often represented with n*8 bytes
+ return Arrays.asList(56, 64, 112, 128, 168, 192);
+ case "DES":
+ return Arrays.asList(56, 64);
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ /** These ciphers can have arbitrary length keys but that's a really bad idea, {@see http://crypto.stackexchange.com/a/9963/12569}.
+ * Also, RC* is deprecated and should be considered insecure */
+ for (int i = 40; i <= 2048; i++) {
+ validKeyLengths.add(i);
+ }
+ return validKeyLengths;
+ case "AES":
+ case "TWOFISH":
+ return Arrays.asList(128, 192, 256);
+ default:
+ return validKeyLengths;
+ }
+ }
+
+ private static int getDefaultKeyLengthForCipher(String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return -1;
+ }
+ cipher = cipher.toUpperCase();
+ switch (cipher) {
+ case "DESEDE":
+ return 112;
+ case "DES":
+ return 64;
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ default:
+ return 128;
+ }
+ }
+
+ public static void processStreams(Cipher cipher, InputStream in, OutputStream out) {
+ try {
+ final byte[] buffer = new byte[BUFFER_SIZE];
+ int len;
+ while ((len = in.read(buffer)) > 0) {
+ final byte[] decryptedBytes = cipher.update(buffer, 0, len);
+ if (decryptedBytes != null) {
+ out.write(decryptedBytes);
+ }
+ }
+
+ try {
--- End diff --
Good catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51914845
--- Diff: nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyDerivationFunction.java ---
@@ -25,8 +25,11 @@
public enum KeyDerivationFunction {
NIFI_LEGACY("NiFi legacy KDF", "MD5 @ 1000 iterations"),
- OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration MD5 compatible with PKCS#5 v1.5");
- // TODO: Implement bcrypt, scrypt, and PBKDF2
+ OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration MD5 compatible with PKCS#5 v1.5"),
+ BCRYPT("Bcrypt", "Bcrypt with configurable work factor: see https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations"),
--- End diff --
Working on that now. Converting various markup languages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the pull request:
https://github.com/apache/nifi/pull/201#issuecomment-178369880
The Travis CI failure is due to my use of `java.util.Base64` which was introduced in Java 8. Working on switching out to `org.apache.commons.codec.binary.Base64` but this does not provide an option to encode without padding unless using URL-safe character set (differing in "+" and "/" -> "-" and "_"). This causes compatibility issues in the Scrypt and Bcrypt salt formats.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51919714
--- Diff: nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/EncryptionMethod.java ---
@@ -26,37 +26,43 @@
*/
public enum EncryptionMethod {
- MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false),
- MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", false),
- MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", false),
- MD5_DES("PBEWITHMD5ANDDES", "BC", false),
- MD5_RC2("PBEWITHMD5ANDRC2", "BC", false),
- SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false),
- SHA1_DES("PBEWITHSHA1ANDDES", "BC", false),
- SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", true),
- SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true),
- SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true),
- SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", true),
- SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", true),
- SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", true),
- SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", true),
- SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", true),
- SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true),
- SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true),
- SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", true),
- SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", true),
- SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", true),
- PGP("PGP", "BC", false),
- PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false);
+ MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false, false),
+ MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", true, false),
+ MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", true, false),
+ MD5_DES("PBEWITHMD5ANDDES", "BC", false, false),
+ MD5_RC2("PBEWITHMD5ANDRC2", "BC", false, false),
+ SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false, false),
+ SHA1_DES("PBEWITHSHA1ANDDES", "BC", false, false),
+ SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", false, false),
+ SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true, false),
+ SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true, false),
+ SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", false, false),
+ SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", false, false),
+ SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", false, false),
+ SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", false, false),
+ SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", false, false),
+ SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true, false),
+ SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true, false),
+ SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", false, false),
+ SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", false, false),
+ SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", false, false),
+ PGP("PGP", "BC", false, false),
+ PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false, false),
+ // New encryption methods which used keyed encryption
+ AES_CBC("AES/CBC/PKCS7Padding", "BC", false, true),
+ AES_CTR("AES/CTR/NoPadding", "BC", false, true),
+ AES_GCM("AES/GCM/NoPadding", "BC", false, true);
private final String algorithm;
private final String provider;
private final boolean unlimitedStrength;
+ private final boolean compatibleWithStrongKDFs;
- EncryptionMethod(String algorithm, String provider, boolean unlimitedStrength) {
+ EncryptionMethod(String algorithm, String provider, boolean unlimitedStrength, boolean compatibleWithStrongKDFs) {
--- End diff --
Can this be a straight override or do we need to keep the original method and deprecate it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the pull request:
https://github.com/apache/nifi/pull/201#issuecomment-179914695
@alopresto I'm running into issues with executing the built assembly without the JCE USJ files installed:
> 2016-02-04 10:52:14,023 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.controller.ControllerService: Provider org.apache.nifi.ssl.StandardSSLContextService could not be instantiated
java.util.ServiceConfigurationError: org.apache.nifi.controller.ControllerService: Provider org.apache.nifi.ssl.StandardSSLContextService could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[na:1.8.0_60]
at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[na:1.8.0_60]
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[na:1.8.0_60]
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[na:1.8.0_60]
at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[na:1.8.0_60]
at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:107) ~[nifi-nar-utils-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:88) ~[nifi-nar-utils-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at org.apache.nifi.NiFi.<init>(NiFi.java:120) ~[nifi-runtime-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at org.apache.nifi.NiFi.main(NiFi.java:227) ~[nifi-runtime-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
Caused by: java.lang.ExceptionInInitializerError: null
at javax.crypto.JceSecurityManager.<clinit>(JceSecurityManager.java:65) ~[na:1.8.0_60]
at javax.crypto.Cipher.getConfiguredPermission(Cipher.java:2587) ~[na:1.8.0_60]
at javax.crypto.Cipher.getMaxAllowedKeyLength(Cipher.java:2611) ~[na:1.8.0_60]
at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:548) ~[na:1.8.0_60]
at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:527) ~[na:1.8.0_60]
at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:194) ~[na:1.8.0_60]
at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:346) ~[na:1.8.0_60]
at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:297) ~[na:1.8.0_60]
at sun.security.ssl.SSLEngineImpl.init(SSLEngineImpl.java:402) ~[na:1.8.0_60]
at sun.security.ssl.SSLEngineImpl.<init>(SSLEngineImpl.java:349) ~[na:1.8.0_60]
at sun.security.ssl.SSLContextImpl.engineCreateSSLEngine(SSLContextImpl.java:201) ~[na:1.8.0_60]
at javax.net.ssl.SSLContext.createSSLEngine(SSLContext.java:329) ~[na:1.8.0_60]
at org.apache.nifi.ssl.StandardSSLContextService.buildAlgorithmAllowableValues(StandardSSLContextService.java:424) ~[nifi-ssl-context-service-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at org.apache.nifi.ssl.StandardSSLContextService.<clinit>(StandardSSLContextService.java:103) ~[nifi-ssl-context-service-0.4.2-SNAPSHOT.jar:0.4.2-SNAPSHOT]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_60]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_60]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_60]
at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[na:1.8.0_60]
at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_60]
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[na:1.8.0_60]
... 6 common frames omitted
Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism
at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:89) ~[na:1.8.0_60]
... 26 common frames omitted
Caused by: java.lang.SecurityException: Cannot locate policy or framework files!
at javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:256) ~[na:1.8.0_60]
at javax.crypto.JceSecurity.access$000(JceSecurity.java:48) ~[na:1.8.0_60]
at javax.crypto.JceSecurity$1.run(JceSecurity.java:81) ~[na:1.8.0_60]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_60]
at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:78) ~[na:1.8.0_60]
... 26 common frames omitted
With them installed, NiFi starts and works as anticipated. Will dig in a bit more to see what changed to cause the issue within
at org.apache.nifi.ssl.StandardSSLContextService.buildAlgorithmAllowableValues(StandardSSLContextService.java:424)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51627413
--- Diff: nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyDerivationFunction.java ---
@@ -44,6 +47,10 @@ public String getDescription() {
return description;
}
+ public boolean isStrongKDF() {
+ return (name.equalsIgnoreCase(BCRYPT.name) || name.equals(SCRYPT.name) || name.equals(PBKDF2.name));
--- End diff --
is there a reason for checking equalsIgnoreCase for BCRYPT but just equals for SCRIPT, PBKDF2?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by mcgilman <gi...@git.apache.org>.
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51630553
--- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/pom.xml ---
@@ -124,4 +124,92 @@
<scope>provided</scope>
</dependency>
</dependencies>
+ <profiles>
+ <profile>
+ <!-- Custom profile for Groovy tests -->
+ <id>groovyUnitTest</id>
--- End diff --
Do we need this profile with it already included in the root pom? Or if there something different about this one? Nit pick, existing profile id's naming convention ... groovy-unit-test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51632415
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/CipherUtility.java ---
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import javax.crypto.Cipher;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class CipherUtility {
+
+ public static final int BUFFER_SIZE = 65536;
+
+ /**
+ * Returns the cipher algorithm from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> AES
+ *
+ * @param algorithm the full algorithm name
+ * @return the generic cipher name or the full algorithm if one cannot be extracted
+ */
+ public static String parseCipherFromAlgorithm(final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return algorithm;
+ }
+ String formattedAlgorithm = algorithm.toUpperCase();
+
+ // This is not optimal but the algorithms do not have a standard format
+ final String AES = "AES";
+ final String TDES = "TRIPLEDES";
+ final String TDES_ALTERNATE = "DESEDE";
+ final String DES = "DES";
+ final String RC4 = "RC4";
+ final String RC2 = "RC2";
+ final String TWOFISH = "TWOFISH";
+ final List<String> SYMMETRIC_CIPHERS = Arrays.asList(AES, TDES, TDES_ALTERNATE, DES, RC4, RC2, TWOFISH);
+
+ // The algorithms contain "TRIPLEDES" but the cipher name is "DESede"
+ final String ACTUAL_TDES_CIPHER = "DESede";
+
+ for (String cipher : SYMMETRIC_CIPHERS) {
+ if (formattedAlgorithm.contains(cipher)) {
+ if (cipher.equals(TDES) || cipher.equals(TDES_ALTERNATE)) {
+ return ACTUAL_TDES_CIPHER;
+ } else {
+ return cipher;
+ }
+ }
+ }
+
+ return algorithm;
+ }
+
+ /**
+ * Returns the cipher key length from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> 128
+ *
+ * @param algorithm the full algorithm name
+ * @return the key length or -1 if one cannot be extracted
+ */
+ public static int parseKeyLengthFromAlgorithm(final String algorithm) {
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ return keyLength;
+ } else {
+ // Key length not explicitly named in algorithm
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ return getDefaultKeyLengthForCipher(cipher);
+ }
+ }
+
+ private static int parseActualKeyLengthFromAlgorithm(final String algorithm) {
+ Pattern pattern = Pattern.compile("([\\d]+)BIT");
+ Matcher matcher = pattern.matcher(algorithm);
+ if (matcher.find()) {
+ return Integer.parseInt(matcher.group(1));
+ } else {
+ return -1;
+ }
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided cipher family. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * Does not reflect if the key length is correct for a specific combination of cipher and PBE-derived key length.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}. However, this method will return {@code true} for both because it only gets the cipher
+ * family, {@code AES}.
+ * <p/>
+ * 64, AES -> false
+ * [128, 192, 256], AES -> true
+ *
+ * @param keyLength the key length in bits
+ * @param cipher the cipher family
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLength(int keyLength, final String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(cipher).contains(keyLength);
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided algorithm. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}.
+ * <p/>
+ * 64, AES/CBC/PKCS7Padding -> false
+ * [128, 192, 256], AES/CBC/PKCS7Padding -> true
+ * <p/>
+ * 128, PBEWITHMD5AND128BITAES-CBC-OPENSSL -> true
+ * [192, 256], PBEWITHMD5AND128BITAES-CBC-OPENSSL -> false
+ *
+ * @param keyLength the key length in bits
+ * @param algorithm the specific algorithm
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLengthForAlgorithm(int keyLength, final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(algorithm).contains(keyLength);
+ }
+
+ public static List<Integer> getValidKeyLengthsForAlgorithm(String algorithm) {
+ List<Integer> validKeyLengths = new ArrayList<>();
+ if (StringUtils.isEmpty(algorithm)) {
+ return validKeyLengths;
+ }
+
+ // Some algorithms specify a single key size
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ validKeyLengths.add(keyLength);
+ return validKeyLengths;
+ }
+
+ // The algorithm does not specify a key size
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ switch (cipher.toUpperCase()) {
+ case "DESEDE":
+ // 3DES keys have the cryptographic strength of 7/8 because of parity bits, but are often represented with n*8 bytes
+ return Arrays.asList(56, 64, 112, 128, 168, 192);
+ case "DES":
+ return Arrays.asList(56, 64);
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ /** These ciphers can have arbitrary length keys but that's a really bad idea, {@see http://crypto.stackexchange.com/a/9963/12569}.
+ * Also, RC* is deprecated and should be considered insecure */
+ for (int i = 40; i <= 2048; i++) {
+ validKeyLengths.add(i);
+ }
+ return validKeyLengths;
+ case "AES":
+ case "TWOFISH":
+ return Arrays.asList(128, 192, 256);
+ default:
+ return validKeyLengths;
+ }
+ }
+
+ private static int getDefaultKeyLengthForCipher(String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return -1;
+ }
+ cipher = cipher.toUpperCase();
+ switch (cipher) {
+ case "DESEDE":
+ return 112;
+ case "DES":
+ return 64;
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ default:
+ return 128;
+ }
+ }
+
+ public static void processStreams(Cipher cipher, InputStream in, OutputStream out) {
+ try {
+ final byte[] buffer = new byte[BUFFER_SIZE];
+ int len;
+ while ((len = in.read(buffer)) > 0) {
+ final byte[] decryptedBytes = cipher.update(buffer, 0, len);
+ if (decryptedBytes != null) {
+ out.write(decryptedBytes);
+ }
+ }
+
+ try {
--- End diff --
Can remove this try/catch, as the handling of Exception is the same as the outer try/catch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51632169
--- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/pom.xml ---
@@ -124,4 +124,92 @@
<scope>provided</scope>
</dependency>
</dependencies>
+ <profiles>
+ <profile>
+ <!-- Custom profile for Groovy tests -->
+ <id>groovyUnitTest</id>
--- End diff --
No, this is an artifact because I was using Groovy tests internally before [NIFI-1365] was accepted. Will remove.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51925068
--- Diff: nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/EncryptionMethod.java ---
@@ -26,37 +26,43 @@
*/
public enum EncryptionMethod {
- MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false),
- MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", false),
- MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", false),
- MD5_DES("PBEWITHMD5ANDDES", "BC", false),
- MD5_RC2("PBEWITHMD5ANDRC2", "BC", false),
- SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false),
- SHA1_DES("PBEWITHSHA1ANDDES", "BC", false),
- SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", true),
- SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true),
- SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true),
- SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", true),
- SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", true),
- SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", true),
- SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", true),
- SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", true),
- SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true),
- SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true),
- SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", true),
- SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", true),
- SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", true),
- PGP("PGP", "BC", false),
- PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false);
+ MD5_128AES("PBEWITHMD5AND128BITAES-CBC-OPENSSL", "BC", false, false),
+ MD5_192AES("PBEWITHMD5AND192BITAES-CBC-OPENSSL", "BC", true, false),
+ MD5_256AES("PBEWITHMD5AND256BITAES-CBC-OPENSSL", "BC", true, false),
+ MD5_DES("PBEWITHMD5ANDDES", "BC", false, false),
+ MD5_RC2("PBEWITHMD5ANDRC2", "BC", false, false),
+ SHA1_RC2("PBEWITHSHA1ANDRC2", "BC", false, false),
+ SHA1_DES("PBEWITHSHA1ANDDES", "BC", false, false),
+ SHA_128AES("PBEWITHSHAAND128BITAES-CBC-BC", "BC", false, false),
+ SHA_192AES("PBEWITHSHAAND192BITAES-CBC-BC", "BC", true, false),
+ SHA_256AES("PBEWITHSHAAND256BITAES-CBC-BC", "BC", true, false),
+ SHA_40RC2("PBEWITHSHAAND40BITRC2-CBC", "BC", false, false),
+ SHA_128RC2("PBEWITHSHAAND128BITRC2-CBC", "BC", false, false),
+ SHA_40RC4("PBEWITHSHAAND40BITRC4", "BC", false, false),
+ SHA_128RC4("PBEWITHSHAAND128BITRC4", "BC", false, false),
+ SHA256_128AES("PBEWITHSHA256AND128BITAES-CBC-BC", "BC", false, false),
+ SHA256_192AES("PBEWITHSHA256AND192BITAES-CBC-BC", "BC", true, false),
+ SHA256_256AES("PBEWITHSHA256AND256BITAES-CBC-BC", "BC", true, false),
+ SHA_2KEYTRIPLEDES("PBEWITHSHAAND2-KEYTRIPLEDES-CBC", "BC", false, false),
+ SHA_3KEYTRIPLEDES("PBEWITHSHAAND3-KEYTRIPLEDES-CBC", "BC", false, false),
+ SHA_TWOFISH("PBEWITHSHAANDTWOFISH-CBC", "BC", false, false),
+ PGP("PGP", "BC", false, false),
+ PGP_ASCII_ARMOR("PGP-ASCII-ARMOR", "BC", false, false),
+ // New encryption methods which used keyed encryption
+ AES_CBC("AES/CBC/PKCS7Padding", "BC", false, true),
+ AES_CTR("AES/CTR/NoPadding", "BC", false, true),
+ AES_GCM("AES/GCM/NoPadding", "BC", false, true);
private final String algorithm;
private final String provider;
private final boolean unlimitedStrength;
+ private final boolean compatibleWithStrongKDFs;
- EncryptionMethod(String algorithm, String provider, boolean unlimitedStrength) {
+ EncryptionMethod(String algorithm, String provider, boolean unlimitedStrength, boolean compatibleWithStrongKDFs) {
--- End diff --
Because this is the enum constructor, it's never used anywhere else.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51930434
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/util/crypto/CipherUtilityGroovyTest.groovy ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto
+
+import org.apache.nifi.security.util.EncryptionMethod
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class CipherUtilityGroovyTest extends GroovyTestCase {
+ private static final Logger logger = LoggerFactory.getLogger(CipherUtilityGroovyTest.class)
+
+ // TripleDES must precede DES for automatic grouping precedence
+ private static final List<String> CIPHERS = ["AES", "TRIPLEDES", "DES", "RC2", "RC4", "RC5", "TWOFISH"]
+ private static final List<String> SYMMETRIC_ALGORITHMS = EncryptionMethod.values().findAll { it.algorithm.startsWith("PBE") || it.algorithm.startsWith("AES") }*.algorithm
+ private static final Map<String, List<String>> ALGORITHMS_MAPPED_BY_CIPHER = SYMMETRIC_ALGORITHMS.groupBy { String algorithm -> CIPHERS.find { algorithm.contains(it) } }
+
+ // Manually mapped as of 01/19/16 0.5.0
+ private static final Map<Integer, List<String>> ALGORITHMS_MAPPED_BY_KEY_LENGTH = [
+ (40) : ["PBEWITHSHAAND40BITRC2-CBC",
+ "PBEWITHSHAAND40BITRC4"],
+ (64) : ["PBEWITHMD5ANDDES",
+ "PBEWITHSHA1ANDDES"],
+ (112): ["PBEWITHSHAAND2-KEYTRIPLEDES-CBC",
+ "PBEWITHSHAAND3-KEYTRIPLEDES-CBC"],
+ (128): ["PBEWITHMD5AND128BITAES-CBC-OPENSSL",
+ "PBEWITHMD5ANDRC2",
+ "PBEWITHSHA1ANDRC2",
+ "PBEWITHSHA256AND128BITAES-CBC-BC",
+ "PBEWITHSHAAND128BITAES-CBC-BC",
+ "PBEWITHSHAAND128BITRC2-CBC",
+ "PBEWITHSHAAND128BITRC4",
+ "PBEWITHSHAANDTWOFISH-CBC",
+ "AES/CBC/PKCS7Padding",
+ "AES/CTR/NoPadding",
+ "AES/GCM/NoPadding"],
+ (192): ["PBEWITHMD5AND192BITAES-CBC-OPENSSL",
+ "PBEWITHSHA256AND192BITAES-CBC-BC",
+ "PBEWITHSHAAND192BITAES-CBC-BC",
+ "AES/CBC/PKCS7Padding",
+ "AES/CTR/NoPadding",
+ "AES/GCM/NoPadding"],
+ (256): ["PBEWITHMD5AND256BITAES-CBC-OPENSSL",
+ "PBEWITHSHA256AND256BITAES-CBC-BC",
+ "PBEWITHSHAAND256BITAES-CBC-BC",
+ "AES/CBC/PKCS7Padding",
+ "AES/CTR/NoPadding",
+ "AES/GCM/NoPadding"]
+ ]
+
+ @BeforeClass
+ static void setUpOnce() {
+ Security.addProvider(new BouncyCastleProvider());
+
+ // Fix because TRIPLEDES -> DESede
+ def tripleDESAlgorithms = ALGORITHMS_MAPPED_BY_CIPHER.remove("TRIPLEDES")
+ ALGORITHMS_MAPPED_BY_CIPHER.put("DESede", tripleDESAlgorithms)
+
+ logger.info("Mapped algorithms: ${ALGORITHMS_MAPPED_BY_CIPHER}")
+ }
+
+ @Before
+ void setUp() throws Exception {
+
+ }
+
+ @After
+ void tearDown() throws Exception {
+
+ }
+
+ @Test
+ void testShouldParseCipherFromAlgorithm() {
+ // Arrange
+ final def EXPECTED_ALGORITHMS = ALGORITHMS_MAPPED_BY_CIPHER
+
+ // Act
+ SYMMETRIC_ALGORITHMS.each { String algorithm ->
+ String cipher = CipherUtility.parseCipherFromAlgorithm(algorithm)
+ logger.info("Extracted ${cipher} from ${algorithm}")
+
+ // Assert
+ assert EXPECTED_ALGORITHMS.get(cipher).contains(algorithm)
+ }
+ }
+
+ @Test
+ void testShouldParseKeyLengthFromAlgorithm() {
+ // Arrange
+ final def EXPECTED_ALGORITHMS = ALGORITHMS_MAPPED_BY_KEY_LENGTH
+
+ // Act
+ SYMMETRIC_ALGORITHMS.each { String algorithm ->
+ int keyLength = CipherUtility.parseKeyLengthFromAlgorithm(algorithm)
+ logger.info("Extracted ${keyLength} from ${algorithm}")
+
+ // Assert
+ assert EXPECTED_ALGORITHMS.get(keyLength).contains(algorithm)
+ }
+ }
+
+ @Test
+ void testShouldDetermineValidKeyLength() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ logger.info("Checking ${keyLength} for ${algorithm}")
+
+ // Assert
+ assert CipherUtility.isValidKeyLength(keyLength, CipherUtility.parseCipherFromAlgorithm(algorithm))
+ }
+ }
+ }
+
+ @Test
+ void testShouldDetermineInvalidKeyLength() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ def invalidKeyLengths = [-1, 0, 1]
+ if (algorithm =~ "RC\\d") {
+ invalidKeyLengths += [39, 2049]
+ } else {
+ invalidKeyLengths += keyLength + 1
+ }
+ logger.info("Checking ${invalidKeyLengths.join(", ")} for ${algorithm}")
+
+ // Assert
+ invalidKeyLengths.each { int invalidKeyLength ->
+ assert !CipherUtility.isValidKeyLength(invalidKeyLength, CipherUtility.parseCipherFromAlgorithm(algorithm))
+ }
+ }
+ }
+ }
+
+ @Test
+ void testShouldDetermineValidKeyLengthForAlgorithm() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ logger.info("Checking ${keyLength} for ${algorithm}")
+
+ // Assert
+ assert CipherUtility.isValidKeyLengthForAlgorithm(keyLength, algorithm)
+ }
+ }
+ }
+
+ @Test
+ void testShouldDetermineInvalidKeyLengthForAlgorithm() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ def invalidKeyLengths = [-1, 0, 1]
+ if (algorithm =~ "RC\\d") {
+ invalidKeyLengths += [39, 2049]
+ } else {
+ invalidKeyLengths += keyLength + 1
+ }
+ logger.info("Checking ${invalidKeyLengths.join(", ")} for ${algorithm}")
+
+ // Assert
+ invalidKeyLengths.each { int invalidKeyLength ->
+ assert !CipherUtility.isValidKeyLengthForAlgorithm(invalidKeyLength, algorithm)
+ }
+ }
+ }
+
+ // Extra hard-coded checks
+ ["PBEWITHSHA256AND256BITAES-CBC-BC": 192].each { String algorithm, int invalidKeyLength ->
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51628287
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java ---
@@ -68,70 +73,78 @@
public static final String DECRYPT_MODE = "Decrypt";
public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
- .name("Mode")
- .description("Specifies whether the content should be encrypted or decrypted")
- .required(true)
- .allowableValues(ENCRYPT_MODE, DECRYPT_MODE)
- .defaultValue(ENCRYPT_MODE)
- .build();
+ .name("Mode")
+ .description("Specifies whether the content should be encrypted or decrypted")
+ .required(true)
+ .allowableValues(ENCRYPT_MODE, DECRYPT_MODE)
+ .defaultValue(ENCRYPT_MODE)
+ .build();
public static final PropertyDescriptor KEY_DERIVATION_FUNCTION = new PropertyDescriptor.Builder()
- .name("key-derivation-function")
- .displayName("Key Derivation Function")
- .description("Specifies the key derivation function to generate the key from the password (and salt)")
- .required(true)
- .allowableValues(KeyDerivationFunction.values())
- .defaultValue(KeyDerivationFunction.NIFI_LEGACY.name())
- .build();
+ .name("key-derivation-function")
+ .displayName("Key Derivation Function")
+ .description("Specifies the key derivation function to generate the key from the password (and salt)")
+ .required(true)
+ .allowableValues(buildKeyDerivationFunctionAllowableValues())
+ .defaultValue(KeyDerivationFunction.NIFI_LEGACY.name())
+ .build();
public static final PropertyDescriptor ENCRYPTION_ALGORITHM = new PropertyDescriptor.Builder()
- .name("Encryption Algorithm")
- .description("The Encryption Algorithm to use")
- .required(true)
- .allowableValues(EncryptionMethod.values())
- .defaultValue(EncryptionMethod.MD5_128AES.name())
- .build();
+ .name("Encryption Algorithm")
+ .description("The Encryption Algorithm to use")
+ .required(true)
+ .allowableValues(buildEncryptionMethodAllowableValues())
+ .defaultValue(EncryptionMethod.MD5_128AES.name())
+ .build();
public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
- .name("Password")
- .description("The Password to use for encrypting or decrypting the data")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .sensitive(true)
- .build();
+ .name("Password")
+ .description("The Password to use for encrypting or decrypting the data")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
public static final PropertyDescriptor PUBLIC_KEYRING = new PropertyDescriptor.Builder()
- .name("public-keyring-file")
- .displayName("Public Keyring File")
- .description("In a PGP encrypt mode, this keyring contains the public key of the recipient")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
+ .name("public-keyring-file")
+ .displayName("Public Keyring File")
+ .description("In a PGP encrypt mode, this keyring contains the public key of the recipient")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
public static final PropertyDescriptor PUBLIC_KEY_USERID = new PropertyDescriptor.Builder()
- .name("public-key-user-id")
- .displayName("Public Key User Id")
- .description("In a PGP encrypt mode, this user id of the recipient")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
+ .name("public-key-user-id")
+ .displayName("Public Key User Id")
+ .description("In a PGP encrypt mode, this user id of the recipient")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
public static final PropertyDescriptor PRIVATE_KEYRING = new PropertyDescriptor.Builder()
- .name("private-keyring-file")
- .displayName("Private Keyring File")
- .description("In a PGP decrypt mode, this keyring contains the private key of the recipient")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
+ .name("private-keyring-file")
+ .displayName("Private Keyring File")
+ .description("In a PGP decrypt mode, this keyring contains the private key of the recipient")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
public static final PropertyDescriptor PRIVATE_KEYRING_PASSPHRASE = new PropertyDescriptor.Builder()
- .name("private-keyring-passphrase")
- .displayName("Private Keyring Passphrase")
- .description("In a PGP decrypt mode, this is the private keyring passphrase")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .sensitive(true)
- .build();
-
+ .name("private-keyring-passphrase")
+ .displayName("Private Keyring Passphrase")
+ .description("In a PGP decrypt mode, this is the private keyring passphrase")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor RAW_KEY_HEX = new PropertyDescriptor.Builder()
+ .name("raw-key-hex")
+ .displayName("Raw key (hexadecimal)")
--- End diff --
I copied the casing and style for `name` from `private-keyring-passphrase` which was an existing property descriptor. Or were you talking about the `displayName` attribute? I will fix that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51631673
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/CipherUtility.java ---
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import javax.crypto.Cipher;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class CipherUtility {
+
+ public static final int BUFFER_SIZE = 65536;
+
+ /**
+ * Returns the cipher algorithm from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> AES
+ *
+ * @param algorithm the full algorithm name
+ * @return the generic cipher name or the full algorithm if one cannot be extracted
+ */
+ public static String parseCipherFromAlgorithm(final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return algorithm;
+ }
+ String formattedAlgorithm = algorithm.toUpperCase();
+
+ // This is not optimal but the algorithms do not have a standard format
+ final String AES = "AES";
+ final String TDES = "TRIPLEDES";
+ final String TDES_ALTERNATE = "DESEDE";
+ final String DES = "DES";
+ final String RC4 = "RC4";
+ final String RC2 = "RC2";
+ final String TWOFISH = "TWOFISH";
+ final List<String> SYMMETRIC_CIPHERS = Arrays.asList(AES, TDES, TDES_ALTERNATE, DES, RC4, RC2, TWOFISH);
+
+ // The algorithms contain "TRIPLEDES" but the cipher name is "DESede"
+ final String ACTUAL_TDES_CIPHER = "DESede";
+
+ for (String cipher : SYMMETRIC_CIPHERS) {
+ if (formattedAlgorithm.contains(cipher)) {
+ if (cipher.equals(TDES) || cipher.equals(TDES_ALTERNATE)) {
+ return ACTUAL_TDES_CIPHER;
+ } else {
+ return cipher;
+ }
+ }
+ }
+
+ return algorithm;
+ }
+
+ /**
+ * Returns the cipher key length from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> 128
+ *
+ * @param algorithm the full algorithm name
+ * @return the key length or -1 if one cannot be extracted
+ */
+ public static int parseKeyLengthFromAlgorithm(final String algorithm) {
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ return keyLength;
+ } else {
+ // Key length not explicitly named in algorithm
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ return getDefaultKeyLengthForCipher(cipher);
+ }
+ }
+
+ private static int parseActualKeyLengthFromAlgorithm(final String algorithm) {
+ Pattern pattern = Pattern.compile("([\\d]+)BIT");
--- End diff --
Would recommend defining the Pattern as a 'private static final' member variable, as it is thread-safe and doesn't change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the pull request:
https://github.com/apache/nifi/pull/201#issuecomment-179947088
Worked with Aldrin and determined that the `brew` method of installing/uninstalling JCE USJ files is not correct, as "uninstalling" removes the unlimited jurisdiction policy files but does not replace them with the originals.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51628311
--- Diff: nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyDerivationFunction.java ---
@@ -44,6 +47,10 @@ public String getDescription() {
return description;
}
+ public boolean isStrongKDF() {
+ return (name.equalsIgnoreCase(BCRYPT.name) || name.equals(SCRYPT.name) || name.equals(PBKDF2.name));
--- End diff --
Nope, will fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the pull request:
https://github.com/apache/nifi/pull/201#issuecomment-179531167
Reviewing latest. Issue with build but only for one JDK, contrib and such look fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51638693
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/CipherUtility.java ---
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import javax.crypto.Cipher;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class CipherUtility {
+
+ public static final int BUFFER_SIZE = 65536;
+
+ /**
+ * Returns the cipher algorithm from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> AES
+ *
+ * @param algorithm the full algorithm name
+ * @return the generic cipher name or the full algorithm if one cannot be extracted
+ */
+ public static String parseCipherFromAlgorithm(final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return algorithm;
+ }
+ String formattedAlgorithm = algorithm.toUpperCase();
+
+ // This is not optimal but the algorithms do not have a standard format
+ final String AES = "AES";
+ final String TDES = "TRIPLEDES";
+ final String TDES_ALTERNATE = "DESEDE";
+ final String DES = "DES";
+ final String RC4 = "RC4";
+ final String RC2 = "RC2";
+ final String TWOFISH = "TWOFISH";
+ final List<String> SYMMETRIC_CIPHERS = Arrays.asList(AES, TDES, TDES_ALTERNATE, DES, RC4, RC2, TWOFISH);
+
+ // The algorithms contain "TRIPLEDES" but the cipher name is "DESede"
+ final String ACTUAL_TDES_CIPHER = "DESede";
+
+ for (String cipher : SYMMETRIC_CIPHERS) {
+ if (formattedAlgorithm.contains(cipher)) {
+ if (cipher.equals(TDES) || cipher.equals(TDES_ALTERNATE)) {
+ return ACTUAL_TDES_CIPHER;
+ } else {
+ return cipher;
+ }
+ }
+ }
+
+ return algorithm;
+ }
+
+ /**
+ * Returns the cipher key length from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> 128
+ *
+ * @param algorithm the full algorithm name
+ * @return the key length or -1 if one cannot be extracted
+ */
+ public static int parseKeyLengthFromAlgorithm(final String algorithm) {
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ return keyLength;
+ } else {
+ // Key length not explicitly named in algorithm
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ return getDefaultKeyLengthForCipher(cipher);
+ }
+ }
+
+ private static int parseActualKeyLengthFromAlgorithm(final String algorithm) {
+ Pattern pattern = Pattern.compile("([\\d]+)BIT");
+ Matcher matcher = pattern.matcher(algorithm);
+ if (matcher.find()) {
+ return Integer.parseInt(matcher.group(1));
+ } else {
+ return -1;
+ }
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided cipher family. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * Does not reflect if the key length is correct for a specific combination of cipher and PBE-derived key length.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}. However, this method will return {@code true} for both because it only gets the cipher
+ * family, {@code AES}.
+ * <p/>
+ * 64, AES -> false
+ * [128, 192, 256], AES -> true
+ *
+ * @param keyLength the key length in bits
+ * @param cipher the cipher family
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLength(int keyLength, final String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(cipher).contains(keyLength);
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided algorithm. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}.
+ * <p/>
+ * 64, AES/CBC/PKCS7Padding -> false
+ * [128, 192, 256], AES/CBC/PKCS7Padding -> true
+ * <p/>
+ * 128, PBEWITHMD5AND128BITAES-CBC-OPENSSL -> true
+ * [192, 256], PBEWITHMD5AND128BITAES-CBC-OPENSSL -> false
+ *
+ * @param keyLength the key length in bits
+ * @param algorithm the specific algorithm
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLengthForAlgorithm(int keyLength, final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(algorithm).contains(keyLength);
+ }
+
+ public static List<Integer> getValidKeyLengthsForAlgorithm(String algorithm) {
+ List<Integer> validKeyLengths = new ArrayList<>();
+ if (StringUtils.isEmpty(algorithm)) {
+ return validKeyLengths;
+ }
+
+ // Some algorithms specify a single key size
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ validKeyLengths.add(keyLength);
+ return validKeyLengths;
+ }
+
+ // The algorithm does not specify a key size
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ switch (cipher.toUpperCase()) {
+ case "DESEDE":
+ // 3DES keys have the cryptographic strength of 7/8 because of parity bits, but are often represented with n*8 bytes
+ return Arrays.asList(56, 64, 112, 128, 168, 192);
+ case "DES":
+ return Arrays.asList(56, 64);
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ /** These ciphers can have arbitrary length keys but that's a really bad idea, {@see http://crypto.stackexchange.com/a/9963/12569}.
+ * Also, RC* is deprecated and should be considered insecure */
+ for (int i = 40; i <= 2048; i++) {
+ validKeyLengths.add(i);
+ }
+ return validKeyLengths;
+ case "AES":
+ case "TWOFISH":
+ return Arrays.asList(128, 192, 256);
+ default:
+ return validKeyLengths;
+ }
+ }
+
+ private static int getDefaultKeyLengthForCipher(String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return -1;
+ }
+ cipher = cipher.toUpperCase();
+ switch (cipher) {
+ case "DESEDE":
+ return 112;
+ case "DES":
+ return 64;
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ default:
+ return 128;
+ }
+ }
+
+ public static void processStreams(Cipher cipher, InputStream in, OutputStream out) {
+ try {
+ final byte[] buffer = new byte[BUFFER_SIZE];
+ int len;
+ while ((len = in.read(buffer)) > 0) {
+ final byte[] decryptedBytes = cipher.update(buffer, 0, len);
+ if (decryptedBytes != null) {
+ out.write(decryptedBytes);
+ }
+ }
+
+ try {
+ out.write(cipher.doFinal());
+ } catch (final Exception e) {
+ throw new ProcessException(e);
+ }
+ } catch (Exception e) {
+ throw new ProcessException(e);
+ }
+ }
+
+ public static byte[] readBytesFromInputStream(InputStream in, String label, int limit, int minimum, byte[] delimiter) throws IOException, ProcessException {
+ if (in == null) {
+ throw new IllegalArgumentException("Cannot read " + label + " from null InputStream");
+ }
+
+ // If the value is not detected within the first n bytes, throw an exception
+ in.mark(limit);
+
+ // The first n bytes of the input stream contain the value up to the custom delimiter
+ if (in.available() < minimum) {
--- End diff --
After discussion with Mark, removed this check as (the intention) is redundant given the `StreamUtils.copyExclusive()` call also uses a limit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51632976
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/CipherUtility.java ---
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import javax.crypto.Cipher;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class CipherUtility {
+
+ public static final int BUFFER_SIZE = 65536;
+
+ /**
+ * Returns the cipher algorithm from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> AES
+ *
+ * @param algorithm the full algorithm name
+ * @return the generic cipher name or the full algorithm if one cannot be extracted
+ */
+ public static String parseCipherFromAlgorithm(final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return algorithm;
+ }
+ String formattedAlgorithm = algorithm.toUpperCase();
+
+ // This is not optimal but the algorithms do not have a standard format
+ final String AES = "AES";
+ final String TDES = "TRIPLEDES";
+ final String TDES_ALTERNATE = "DESEDE";
+ final String DES = "DES";
+ final String RC4 = "RC4";
+ final String RC2 = "RC2";
+ final String TWOFISH = "TWOFISH";
+ final List<String> SYMMETRIC_CIPHERS = Arrays.asList(AES, TDES, TDES_ALTERNATE, DES, RC4, RC2, TWOFISH);
+
+ // The algorithms contain "TRIPLEDES" but the cipher name is "DESede"
+ final String ACTUAL_TDES_CIPHER = "DESede";
+
+ for (String cipher : SYMMETRIC_CIPHERS) {
+ if (formattedAlgorithm.contains(cipher)) {
+ if (cipher.equals(TDES) || cipher.equals(TDES_ALTERNATE)) {
+ return ACTUAL_TDES_CIPHER;
+ } else {
+ return cipher;
+ }
+ }
+ }
+
+ return algorithm;
+ }
+
+ /**
+ * Returns the cipher key length from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> 128
+ *
+ * @param algorithm the full algorithm name
+ * @return the key length or -1 if one cannot be extracted
+ */
+ public static int parseKeyLengthFromAlgorithm(final String algorithm) {
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ return keyLength;
+ } else {
+ // Key length not explicitly named in algorithm
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ return getDefaultKeyLengthForCipher(cipher);
+ }
+ }
+
+ private static int parseActualKeyLengthFromAlgorithm(final String algorithm) {
+ Pattern pattern = Pattern.compile("([\\d]+)BIT");
+ Matcher matcher = pattern.matcher(algorithm);
+ if (matcher.find()) {
+ return Integer.parseInt(matcher.group(1));
+ } else {
+ return -1;
+ }
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided cipher family. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * Does not reflect if the key length is correct for a specific combination of cipher and PBE-derived key length.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}. However, this method will return {@code true} for both because it only gets the cipher
+ * family, {@code AES}.
+ * <p/>
+ * 64, AES -> false
+ * [128, 192, 256], AES -> true
+ *
+ * @param keyLength the key length in bits
+ * @param cipher the cipher family
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLength(int keyLength, final String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(cipher).contains(keyLength);
+ }
+
+ /**
+ * Returns true if the provided key length is a valid key length for the provided algorithm. Does not reflect if the Unlimited Strength Cryptography Jurisdiction Policies are installed.
+ * <p/>
+ * Ex:
+ * <p/>
+ * 256 is valid for {@code AES/CBC/PKCS7Padding} but not {@code PBEWITHMD5AND128BITAES-CBC-OPENSSL}.
+ * <p/>
+ * 64, AES/CBC/PKCS7Padding -> false
+ * [128, 192, 256], AES/CBC/PKCS7Padding -> true
+ * <p/>
+ * 128, PBEWITHMD5AND128BITAES-CBC-OPENSSL -> true
+ * [192, 256], PBEWITHMD5AND128BITAES-CBC-OPENSSL -> false
+ *
+ * @param keyLength the key length in bits
+ * @param algorithm the specific algorithm
+ * @return true if this key length is valid
+ */
+ public static boolean isValidKeyLengthForAlgorithm(int keyLength, final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return false;
+ }
+ return getValidKeyLengthsForAlgorithm(algorithm).contains(keyLength);
+ }
+
+ public static List<Integer> getValidKeyLengthsForAlgorithm(String algorithm) {
+ List<Integer> validKeyLengths = new ArrayList<>();
+ if (StringUtils.isEmpty(algorithm)) {
+ return validKeyLengths;
+ }
+
+ // Some algorithms specify a single key size
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ validKeyLengths.add(keyLength);
+ return validKeyLengths;
+ }
+
+ // The algorithm does not specify a key size
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ switch (cipher.toUpperCase()) {
+ case "DESEDE":
+ // 3DES keys have the cryptographic strength of 7/8 because of parity bits, but are often represented with n*8 bytes
+ return Arrays.asList(56, 64, 112, 128, 168, 192);
+ case "DES":
+ return Arrays.asList(56, 64);
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ /** These ciphers can have arbitrary length keys but that's a really bad idea, {@see http://crypto.stackexchange.com/a/9963/12569}.
+ * Also, RC* is deprecated and should be considered insecure */
+ for (int i = 40; i <= 2048; i++) {
+ validKeyLengths.add(i);
+ }
+ return validKeyLengths;
+ case "AES":
+ case "TWOFISH":
+ return Arrays.asList(128, 192, 256);
+ default:
+ return validKeyLengths;
+ }
+ }
+
+ private static int getDefaultKeyLengthForCipher(String cipher) {
+ if (StringUtils.isEmpty(cipher)) {
+ return -1;
+ }
+ cipher = cipher.toUpperCase();
+ switch (cipher) {
+ case "DESEDE":
+ return 112;
+ case "DES":
+ return 64;
+ case "RC2":
+ case "RC4":
+ case "RC5":
+ default:
+ return 128;
+ }
+ }
+
+ public static void processStreams(Cipher cipher, InputStream in, OutputStream out) {
+ try {
+ final byte[] buffer = new byte[BUFFER_SIZE];
+ int len;
+ while ((len = in.read(buffer)) > 0) {
+ final byte[] decryptedBytes = cipher.update(buffer, 0, len);
+ if (decryptedBytes != null) {
+ out.write(decryptedBytes);
+ }
+ }
+
+ try {
+ out.write(cipher.doFinal());
+ } catch (final Exception e) {
+ throw new ProcessException(e);
+ }
+ } catch (Exception e) {
+ throw new ProcessException(e);
+ }
+ }
+
+ public static byte[] readBytesFromInputStream(InputStream in, String label, int limit, int minimum, byte[] delimiter) throws IOException, ProcessException {
+ if (in == null) {
+ throw new IllegalArgumentException("Cannot read " + label + " from null InputStream");
+ }
+
+ // If the value is not detected within the first n bytes, throw an exception
+ in.mark(limit);
+
+ // The first n bytes of the input stream contain the value up to the custom delimiter
+ if (in.available() < minimum) {
--- End diff --
I think this logic may be incorrect - in.available() could return 0 or 1 or any number of values even if the stream has a huge amount of data available to be read, if reading it could result in blocking for I/O to occur...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51632097
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/CipherUtility.java ---
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.ByteArrayOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import javax.crypto.Cipher;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class CipherUtility {
+
+ public static final int BUFFER_SIZE = 65536;
+
+ /**
+ * Returns the cipher algorithm from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> AES
+ *
+ * @param algorithm the full algorithm name
+ * @return the generic cipher name or the full algorithm if one cannot be extracted
+ */
+ public static String parseCipherFromAlgorithm(final String algorithm) {
+ if (StringUtils.isEmpty(algorithm)) {
+ return algorithm;
+ }
+ String formattedAlgorithm = algorithm.toUpperCase();
+
+ // This is not optimal but the algorithms do not have a standard format
+ final String AES = "AES";
+ final String TDES = "TRIPLEDES";
+ final String TDES_ALTERNATE = "DESEDE";
+ final String DES = "DES";
+ final String RC4 = "RC4";
+ final String RC2 = "RC2";
+ final String TWOFISH = "TWOFISH";
+ final List<String> SYMMETRIC_CIPHERS = Arrays.asList(AES, TDES, TDES_ALTERNATE, DES, RC4, RC2, TWOFISH);
+
+ // The algorithms contain "TRIPLEDES" but the cipher name is "DESede"
+ final String ACTUAL_TDES_CIPHER = "DESede";
+
+ for (String cipher : SYMMETRIC_CIPHERS) {
+ if (formattedAlgorithm.contains(cipher)) {
+ if (cipher.equals(TDES) || cipher.equals(TDES_ALTERNATE)) {
+ return ACTUAL_TDES_CIPHER;
+ } else {
+ return cipher;
+ }
+ }
+ }
+
+ return algorithm;
+ }
+
+ /**
+ * Returns the cipher key length from the full algorithm name. Useful for getting key lengths, etc.
+ * <p/>
+ * Ex: PBEWITHMD5AND128BITAES-CBC-OPENSSL -> 128
+ *
+ * @param algorithm the full algorithm name
+ * @return the key length or -1 if one cannot be extracted
+ */
+ public static int parseKeyLengthFromAlgorithm(final String algorithm) {
+ int keyLength = parseActualKeyLengthFromAlgorithm(algorithm);
+ if (keyLength != -1) {
+ return keyLength;
+ } else {
+ // Key length not explicitly named in algorithm
+ String cipher = parseCipherFromAlgorithm(algorithm);
+ return getDefaultKeyLengthForCipher(cipher);
+ }
+ }
+
+ private static int parseActualKeyLengthFromAlgorithm(final String algorithm) {
+ Pattern pattern = Pattern.compile("([\\d]+)BIT");
--- End diff --
Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51930882
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/logback-test.xml ---
@@ -57,11 +60,12 @@
<logger name="org.apache.nifi.processors.standard" level="DEBUG"/>
<logger name="target.file" level="DEBUG" additivity="true">
- <appender-ref ref="TARGET_FILE" />
+ <appender-ref ref="TARGET_FILE"/>
</logger>
- <root level="INFO">
+ <!--<root level="${overrideLog:-DEBUG}">-->
+ <root level="DEBUG">
--- End diff --
I never actually got this to work. Reverting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51925709
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/util/crypto/CipherUtilityGroovyTest.groovy ---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto
+
+import org.apache.nifi.security.util.EncryptionMethod
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class CipherUtilityGroovyTest extends GroovyTestCase {
+ private static final Logger logger = LoggerFactory.getLogger(CipherUtilityGroovyTest.class)
+
+ // TripleDES must precede DES for automatic grouping precedence
+ private static final List<String> CIPHERS = ["AES", "TRIPLEDES", "DES", "RC2", "RC4", "RC5", "TWOFISH"]
+ private static final List<String> SYMMETRIC_ALGORITHMS = EncryptionMethod.values().findAll { it.algorithm.startsWith("PBE") || it.algorithm.startsWith("AES") }*.algorithm
+ private static final Map<String, List<String>> ALGORITHMS_MAPPED_BY_CIPHER = SYMMETRIC_ALGORITHMS.groupBy { String algorithm -> CIPHERS.find { algorithm.contains(it) } }
+
+ // Manually mapped as of 01/19/16 0.5.0
+ private static final Map<Integer, List<String>> ALGORITHMS_MAPPED_BY_KEY_LENGTH = [
+ (40) : ["PBEWITHSHAAND40BITRC2-CBC",
+ "PBEWITHSHAAND40BITRC4"],
+ (64) : ["PBEWITHMD5ANDDES",
+ "PBEWITHSHA1ANDDES"],
+ (112): ["PBEWITHSHAAND2-KEYTRIPLEDES-CBC",
+ "PBEWITHSHAAND3-KEYTRIPLEDES-CBC"],
+ (128): ["PBEWITHMD5AND128BITAES-CBC-OPENSSL",
+ "PBEWITHMD5ANDRC2",
+ "PBEWITHSHA1ANDRC2",
+ "PBEWITHSHA256AND128BITAES-CBC-BC",
+ "PBEWITHSHAAND128BITAES-CBC-BC",
+ "PBEWITHSHAAND128BITRC2-CBC",
+ "PBEWITHSHAAND128BITRC4",
+ "PBEWITHSHAANDTWOFISH-CBC",
+ "AES/CBC/PKCS7Padding",
+ "AES/CTR/NoPadding",
+ "AES/GCM/NoPadding"],
+ (192): ["PBEWITHMD5AND192BITAES-CBC-OPENSSL",
+ "PBEWITHSHA256AND192BITAES-CBC-BC",
+ "PBEWITHSHAAND192BITAES-CBC-BC",
+ "AES/CBC/PKCS7Padding",
+ "AES/CTR/NoPadding",
+ "AES/GCM/NoPadding"],
+ (256): ["PBEWITHMD5AND256BITAES-CBC-OPENSSL",
+ "PBEWITHSHA256AND256BITAES-CBC-BC",
+ "PBEWITHSHAAND256BITAES-CBC-BC",
+ "AES/CBC/PKCS7Padding",
+ "AES/CTR/NoPadding",
+ "AES/GCM/NoPadding"]
+ ]
+
+ @BeforeClass
+ static void setUpOnce() {
+ Security.addProvider(new BouncyCastleProvider());
+
+ // Fix because TRIPLEDES -> DESede
+ def tripleDESAlgorithms = ALGORITHMS_MAPPED_BY_CIPHER.remove("TRIPLEDES")
+ ALGORITHMS_MAPPED_BY_CIPHER.put("DESede", tripleDESAlgorithms)
+
+ logger.info("Mapped algorithms: ${ALGORITHMS_MAPPED_BY_CIPHER}")
+ }
+
+ @Before
+ void setUp() throws Exception {
+
+ }
+
+ @After
+ void tearDown() throws Exception {
+
+ }
+
+ @Test
+ void testShouldParseCipherFromAlgorithm() {
+ // Arrange
+ final def EXPECTED_ALGORITHMS = ALGORITHMS_MAPPED_BY_CIPHER
+
+ // Act
+ SYMMETRIC_ALGORITHMS.each { String algorithm ->
+ String cipher = CipherUtility.parseCipherFromAlgorithm(algorithm)
+ logger.info("Extracted ${cipher} from ${algorithm}")
+
+ // Assert
+ assert EXPECTED_ALGORITHMS.get(cipher).contains(algorithm)
+ }
+ }
+
+ @Test
+ void testShouldParseKeyLengthFromAlgorithm() {
+ // Arrange
+ final def EXPECTED_ALGORITHMS = ALGORITHMS_MAPPED_BY_KEY_LENGTH
+
+ // Act
+ SYMMETRIC_ALGORITHMS.each { String algorithm ->
+ int keyLength = CipherUtility.parseKeyLengthFromAlgorithm(algorithm)
+ logger.info("Extracted ${keyLength} from ${algorithm}")
+
+ // Assert
+ assert EXPECTED_ALGORITHMS.get(keyLength).contains(algorithm)
+ }
+ }
+
+ @Test
+ void testShouldDetermineValidKeyLength() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ logger.info("Checking ${keyLength} for ${algorithm}")
+
+ // Assert
+ assert CipherUtility.isValidKeyLength(keyLength, CipherUtility.parseCipherFromAlgorithm(algorithm))
+ }
+ }
+ }
+
+ @Test
+ void testShouldDetermineInvalidKeyLength() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ def invalidKeyLengths = [-1, 0, 1]
+ if (algorithm =~ "RC\\d") {
+ invalidKeyLengths += [39, 2049]
+ } else {
+ invalidKeyLengths += keyLength + 1
+ }
+ logger.info("Checking ${invalidKeyLengths.join(", ")} for ${algorithm}")
+
+ // Assert
+ invalidKeyLengths.each { int invalidKeyLength ->
+ assert !CipherUtility.isValidKeyLength(invalidKeyLength, CipherUtility.parseCipherFromAlgorithm(algorithm))
+ }
+ }
+ }
+ }
+
+ @Test
+ void testShouldDetermineValidKeyLengthForAlgorithm() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ logger.info("Checking ${keyLength} for ${algorithm}")
+
+ // Assert
+ assert CipherUtility.isValidKeyLengthForAlgorithm(keyLength, algorithm)
+ }
+ }
+ }
+
+ @Test
+ void testShouldDetermineInvalidKeyLengthForAlgorithm() {
+ // Arrange
+
+ // Act
+ ALGORITHMS_MAPPED_BY_KEY_LENGTH.each { int keyLength, List<String> algorithms ->
+ algorithms.each { String algorithm ->
+ def invalidKeyLengths = [-1, 0, 1]
+ if (algorithm =~ "RC\\d") {
+ invalidKeyLengths += [39, 2049]
+ } else {
+ invalidKeyLengths += keyLength + 1
+ }
+ logger.info("Checking ${invalidKeyLengths.join(", ")} for ${algorithm}")
+
+ // Assert
+ invalidKeyLengths.each { int invalidKeyLength ->
+ assert !CipherUtility.isValidKeyLengthForAlgorithm(invalidKeyLength, algorithm)
+ }
+ }
+ }
+
+ // Extra hard-coded checks
+ ["PBEWITHSHA256AND256BITAES-CBC-BC": 192].each { String algorithm, int invalidKeyLength ->
--- End diff --
Let's add extra entries here (or do away with the map)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51639833
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/OpenSSLPKCS5CipherProvider.java ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.security.util.EncryptionMethod;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.PBEParameterSpec;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.util.Arrays;
+
+public class OpenSSLPKCS5CipherProvider implements PBECipherProvider {
+ private static final Logger logger = LoggerFactory.getLogger(OpenSSLPKCS5CipherProvider.class);
+
+ // Legacy magic number value
+ private static final int ITERATION_COUNT = 0;
+ private static final int DEFAULT_SALT_LENGTH = 8;
+ private static final byte[] EMPTY_SALT = new byte[8];
+
+ private static final String OPENSSL_EVP_HEADER_MARKER = "Salted__";
+ private static final int OPENSSL_EVP_HEADER_SIZE = 8;
+
+ /**
+ * Returns an initialized cipher for the specified algorithm. The key (and IV if necessary) are derived using the
+ * <a href="https://www.openssl.org/docs/manmaster/crypto/EVP_BytesToKey.html">OpenSSL EVP_BytesToKey proprietary KDF</a> [essentially {@code MD5(password || salt) }].
+ *
+ * @param encryptionMethod the {@link EncryptionMethod}
+ * @param password the secret input
+ * @param keyLength the desired key length in bits (ignored because OpenSSL ciphers provide key length in algorithm name)
+ * @param encryptMode true for encrypt, false for decrypt
+ * @return the initialized cipher
+ * @throws Exception if there is a problem initializing the cipher
+ */
+ @Override
+ public Cipher getCipher(EncryptionMethod encryptionMethod, String password, int keyLength, boolean encryptMode) throws Exception {
+ return getCipher(encryptionMethod, password, new byte[0], keyLength, encryptMode);
+ }
+
+ /**
+ * Returns an initialized cipher for the specified algorithm. The key (and IV if necessary) are derived using the
+ * <a href="https://www.openssl.org/docs/manmaster/crypto/EVP_BytesToKey.html">OpenSSL EVP_BytesToKey proprietary KDF</a> [essentially {@code MD5(password || salt) }].
+ *
+ * @param encryptionMethod the {@link EncryptionMethod}
+ * @param password the secret input
+ * @param salt the salt
+ * @param keyLength the desired key length in bits (ignored because OpenSSL ciphers provide key length in algorithm name)
+ * @param encryptMode true for encrypt, false for decrypt
+ * @return the initialized cipher
+ * @throws Exception if there is a problem initializing the cipher
+ */
+ @Override
+ public Cipher getCipher(EncryptionMethod encryptionMethod, String password, byte[] salt, int keyLength, boolean encryptMode) throws Exception {
+ try {
+ return getInitializedCipher(encryptionMethod, password, salt, encryptMode);
+ } catch (IllegalArgumentException e) {
--- End diff --
Is there a reason why this IllegalArgumentException is caught in lieu of catching/wrapping this with the associated ProcessException below?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/201
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51914669
--- Diff: nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyDerivationFunction.java ---
@@ -25,8 +25,11 @@
public enum KeyDerivationFunction {
NIFI_LEGACY("NiFi legacy KDF", "MD5 @ 1000 iterations"),
- OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration MD5 compatible with PKCS#5 v1.5");
- // TODO: Implement bcrypt, scrypt, and PBKDF2
+ OPENSSL_EVP_BYTES_TO_KEY("OpenSSL EVP_BytesToKey", "Single iteration MD5 compatible with PKCS#5 v1.5"),
+ BCRYPT("Bcrypt", "Bcrypt with configurable work factor: see https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations"),
--- End diff --
The resources listed on the wiki are super helpful. I think it would make sense for these to additionally get rolled into the administrator's guide so we can have a one stop shop for all of it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51640371
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/OpenSSLPKCS5CipherProvider.java ---
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.util.crypto;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.security.util.EncryptionMethod;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.SecretKeyFactory;
+import javax.crypto.spec.PBEKeySpec;
+import javax.crypto.spec.PBEParameterSpec;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.security.spec.InvalidKeySpecException;
+import java.util.Arrays;
+
+public class OpenSSLPKCS5CipherProvider implements PBECipherProvider {
+ private static final Logger logger = LoggerFactory.getLogger(OpenSSLPKCS5CipherProvider.class);
+
+ // Legacy magic number value
+ private static final int ITERATION_COUNT = 0;
+ private static final int DEFAULT_SALT_LENGTH = 8;
+ private static final byte[] EMPTY_SALT = new byte[8];
+
+ private static final String OPENSSL_EVP_HEADER_MARKER = "Salted__";
+ private static final int OPENSSL_EVP_HEADER_SIZE = 8;
+
+ /**
+ * Returns an initialized cipher for the specified algorithm. The key (and IV if necessary) are derived using the
+ * <a href="https://www.openssl.org/docs/manmaster/crypto/EVP_BytesToKey.html">OpenSSL EVP_BytesToKey proprietary KDF</a> [essentially {@code MD5(password || salt) }].
+ *
+ * @param encryptionMethod the {@link EncryptionMethod}
+ * @param password the secret input
+ * @param keyLength the desired key length in bits (ignored because OpenSSL ciphers provide key length in algorithm name)
+ * @param encryptMode true for encrypt, false for decrypt
+ * @return the initialized cipher
+ * @throws Exception if there is a problem initializing the cipher
+ */
+ @Override
+ public Cipher getCipher(EncryptionMethod encryptionMethod, String password, int keyLength, boolean encryptMode) throws Exception {
+ return getCipher(encryptionMethod, password, new byte[0], keyLength, encryptMode);
+ }
+
+ /**
+ * Returns an initialized cipher for the specified algorithm. The key (and IV if necessary) are derived using the
+ * <a href="https://www.openssl.org/docs/manmaster/crypto/EVP_BytesToKey.html">OpenSSL EVP_BytesToKey proprietary KDF</a> [essentially {@code MD5(password || salt) }].
+ *
+ * @param encryptionMethod the {@link EncryptionMethod}
+ * @param password the secret input
+ * @param salt the salt
+ * @param keyLength the desired key length in bits (ignored because OpenSSL ciphers provide key length in algorithm name)
+ * @param encryptMode true for encrypt, false for decrypt
+ * @return the initialized cipher
+ * @throws Exception if there is a problem initializing the cipher
+ */
+ @Override
+ public Cipher getCipher(EncryptionMethod encryptionMethod, String password, byte[] salt, int keyLength, boolean encryptMode) throws Exception {
+ try {
+ return getInitializedCipher(encryptionMethod, password, salt, encryptMode);
+ } catch (IllegalArgumentException e) {
--- End diff --
During unit testing I was treating these differently. I guess in a production application the validation is not done any earlier, so it may not provide any value. I just figured that with crypto stuff, so much of it is opaque to a user that if I could separate out "you need to provide a good value for _X_" vs. "_magic_ went wrong" exceptions, that would be helpful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the pull request:
https://github.com/apache/nifi/pull/201#issuecomment-179003037
The current state of the code passes on Java 7 and 8 with unlimited strength cryptographic jurisdiction policies installed (Travis includes these). I am resolving some tests which would fail on a vanilla JVM without these policies installed and will push these fixes.
I also discovered some inconsistencies in the `BouncyCastle` implementation of certain PBE ciphers that can exceed the key strength limit even when using the correct key length if the underlying password length exceeds an undocumented inner limit. This was originally recorded in [NIFI-1255](https://issues.apache.org/jira/browse/NIFI-1255) but I have added a custom validator temporarily and added further documentation to that ticket.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51930432
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/logback-test.xml ---
@@ -57,11 +60,12 @@
<logger name="org.apache.nifi.processors.standard" level="DEBUG"/>
<logger name="target.file" level="DEBUG" additivity="true">
- <appender-ref ref="TARGET_FILE" />
+ <appender-ref ref="TARGET_FILE"/>
</logger>
- <root level="INFO">
+ <!--<root level="${overrideLog:-DEBUG}">-->
+ <root level="DEBUG">
--- End diff --
Did you intend to revert this back to the override log variable and default to INFO?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1257 and 1259
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/201#discussion_r51627867
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java ---
@@ -68,70 +73,78 @@
public static final String DECRYPT_MODE = "Decrypt";
public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
- .name("Mode")
- .description("Specifies whether the content should be encrypted or decrypted")
- .required(true)
- .allowableValues(ENCRYPT_MODE, DECRYPT_MODE)
- .defaultValue(ENCRYPT_MODE)
- .build();
+ .name("Mode")
+ .description("Specifies whether the content should be encrypted or decrypted")
+ .required(true)
+ .allowableValues(ENCRYPT_MODE, DECRYPT_MODE)
+ .defaultValue(ENCRYPT_MODE)
+ .build();
public static final PropertyDescriptor KEY_DERIVATION_FUNCTION = new PropertyDescriptor.Builder()
- .name("key-derivation-function")
- .displayName("Key Derivation Function")
- .description("Specifies the key derivation function to generate the key from the password (and salt)")
- .required(true)
- .allowableValues(KeyDerivationFunction.values())
- .defaultValue(KeyDerivationFunction.NIFI_LEGACY.name())
- .build();
+ .name("key-derivation-function")
+ .displayName("Key Derivation Function")
+ .description("Specifies the key derivation function to generate the key from the password (and salt)")
+ .required(true)
+ .allowableValues(buildKeyDerivationFunctionAllowableValues())
+ .defaultValue(KeyDerivationFunction.NIFI_LEGACY.name())
+ .build();
public static final PropertyDescriptor ENCRYPTION_ALGORITHM = new PropertyDescriptor.Builder()
- .name("Encryption Algorithm")
- .description("The Encryption Algorithm to use")
- .required(true)
- .allowableValues(EncryptionMethod.values())
- .defaultValue(EncryptionMethod.MD5_128AES.name())
- .build();
+ .name("Encryption Algorithm")
+ .description("The Encryption Algorithm to use")
+ .required(true)
+ .allowableValues(buildEncryptionMethodAllowableValues())
+ .defaultValue(EncryptionMethod.MD5_128AES.name())
+ .build();
public static final PropertyDescriptor PASSWORD = new PropertyDescriptor.Builder()
- .name("Password")
- .description("The Password to use for encrypting or decrypting the data")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .sensitive(true)
- .build();
+ .name("Password")
+ .description("The Password to use for encrypting or decrypting the data")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
public static final PropertyDescriptor PUBLIC_KEYRING = new PropertyDescriptor.Builder()
- .name("public-keyring-file")
- .displayName("Public Keyring File")
- .description("In a PGP encrypt mode, this keyring contains the public key of the recipient")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
+ .name("public-keyring-file")
+ .displayName("Public Keyring File")
+ .description("In a PGP encrypt mode, this keyring contains the public key of the recipient")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
public static final PropertyDescriptor PUBLIC_KEY_USERID = new PropertyDescriptor.Builder()
- .name("public-key-user-id")
- .displayName("Public Key User Id")
- .description("In a PGP encrypt mode, this user id of the recipient")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
+ .name("public-key-user-id")
+ .displayName("Public Key User Id")
+ .description("In a PGP encrypt mode, this user id of the recipient")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
public static final PropertyDescriptor PRIVATE_KEYRING = new PropertyDescriptor.Builder()
- .name("private-keyring-file")
- .displayName("Private Keyring File")
- .description("In a PGP decrypt mode, this keyring contains the private key of the recipient")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .build();
+ .name("private-keyring-file")
+ .displayName("Private Keyring File")
+ .description("In a PGP decrypt mode, this keyring contains the private key of the recipient")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
public static final PropertyDescriptor PRIVATE_KEYRING_PASSPHRASE = new PropertyDescriptor.Builder()
- .name("private-keyring-passphrase")
- .displayName("Private Keyring Passphrase")
- .description("In a PGP decrypt mode, this is the private keyring passphrase")
- .required(false)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .sensitive(true)
- .build();
-
+ .name("private-keyring-passphrase")
+ .displayName("Private Keyring Passphrase")
+ .description("In a PGP decrypt mode, this is the private keyring passphrase")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .sensitive(true)
+ .build();
+ public static final PropertyDescriptor RAW_KEY_HEX = new PropertyDescriptor.Builder()
+ .name("raw-key-hex")
+ .displayName("Raw key (hexadecimal)")
--- End diff --
Would recommend using Title Case for consistency with other processor properties
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---