You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/06/04 13:03:59 UTC
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5122: NIFI-8499 - encrypted flow repository swap file implementation
exceptionfactory commented on a change in pull request #5122:
URL: https://github.com/apache/nifi/pull/5122#discussion_r645540468
##########
File path: nifi-docs/src/main/asciidoc/administration-guide.adoc
##########
@@ -2961,7 +2961,8 @@ available again. These properties govern how that process occurs.
|====
|*Property*|*Description*
-|`nifi.swap.manager.implementation`|The Swap Manager implementation. The default value is `org.apache.nifi.controller.FileSystemSwapManager` and should not be changed.
+|`nifi.swap.manager.implementation`| The Swap Manager implementation. The default value is `org.apache.nifi.controller.FileSystemSwapManager`.
+There is also an experimental implementation, `org.apache.nifi.controller.FileSystemSwapManagerEncrypt`, that encrypts the swap file content on disk, using the configured FlowFile encryption key.
Review comment:
I don't think it is necessary to label the encrypted version as "experimental". I recommend including some additional details on the encryption algorithm and cipher used in the implementation.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
Review comment:
Each repository can currently have its own encryption key, so expanding the description to include "FlowFile Repository encryption key" would be helpful.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ protected InputStream getInputStream(final File file) throws IOException {
+ final FileInputStream fis = new FileInputStream(file);
+ try {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ final int countIV = fis.read(iv);
+ if (countIV != SIZE_IV_AES_BYTES) {
+ throw new IOException(String.format(
+ "problem reading IV [expected=%d, actual=%d]", SIZE_IV_AES_BYTES, countIV));
+ }
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.DECRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherInputStream(fis, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ protected OutputStream getOutputStream(final File file) throws IOException {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ new SecureRandom().nextBytes(iv);
+ final FileOutputStream fos = new FileOutputStream(file);
+ fos.write(iv);
+ try {
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.ENCRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherOutputStream(fos, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ private static final String ALGORITHM = "AES";
Review comment:
Is this property used?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapManagerInitializationContext;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.StaticKeyProvider;
+import org.apache.nifi.util.NiFiProperties;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.logging.Logger;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test cases for {@link FileSystemSwapManagerEncrypt}.
+ */
+public class TestFileSystemSwapManagerEncrypt {
+ private static final Logger logger = Logger.getLogger(TestFileSystemSwapManagerEncrypt.class.getName());
+
+ /**
+ * Test a simple swap to disk / swap from disk operation. Configured to use {@link StaticKeyProvider}.
+ */
+ @Test
+ public void testConfig_StaticKeyProvider_SerdeSuccess() throws GeneralSecurityException, EncryptionException, IOException {
Review comment:
Recommend avoiding the `_` character in method names. Perhaps something like `testSwapInSwapOut()`?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
Review comment:
Following the convention of the encrypted repositories, recommend renaming this to `EncryptedFileSystemSwapManager`.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ protected InputStream getInputStream(final File file) throws IOException {
+ final FileInputStream fis = new FileInputStream(file);
+ try {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ final int countIV = fis.read(iv);
+ if (countIV != SIZE_IV_AES_BYTES) {
+ throw new IOException(String.format(
+ "problem reading IV [expected=%d, actual=%d]", SIZE_IV_AES_BYTES, countIV));
+ }
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.DECRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherInputStream(fis, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
Review comment:
Recommend including a message with this exception that include the absolute path of the File for debugging purposes.
```suggestion
throw new IOException(String.format("Preparing Cipher Failed for File [%s]", file.getAbsolutePath()), e);
```
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
Review comment:
The implementation appears to use AES-GCM as opposed to AES-CTR as stated in this description.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
Review comment:
Can this constructor be removed?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ protected InputStream getInputStream(final File file) throws IOException {
+ final FileInputStream fis = new FileInputStream(file);
+ try {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ final int countIV = fis.read(iv);
+ if (countIV != SIZE_IV_AES_BYTES) {
+ throw new IOException(String.format(
+ "problem reading IV [expected=%d, actual=%d]", SIZE_IV_AES_BYTES, countIV));
+ }
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.DECRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherInputStream(fis, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ protected OutputStream getOutputStream(final File file) throws IOException {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ new SecureRandom().nextBytes(iv);
+ final FileOutputStream fos = new FileOutputStream(file);
+ fos.write(iv);
+ try {
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.ENCRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherOutputStream(fos, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
Review comment:
Recommend including a message that referenced the file being written.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ protected InputStream getInputStream(final File file) throws IOException {
+ final FileInputStream fis = new FileInputStream(file);
+ try {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ final int countIV = fis.read(iv);
+ if (countIV != SIZE_IV_AES_BYTES) {
+ throw new IOException(String.format(
+ "problem reading IV [expected=%d, actual=%d]", SIZE_IV_AES_BYTES, countIV));
+ }
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.DECRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherInputStream(fis, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ protected OutputStream getOutputStream(final File file) throws IOException {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ new SecureRandom().nextBytes(iv);
+ final FileOutputStream fos = new FileOutputStream(file);
+ fos.write(iv);
+ try {
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.ENCRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherOutputStream(fos, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ private static final String ALGORITHM = "AES";
+ private static final String TRANSFORMATION = "AES/GCM/NoPadding";
Review comment:
Perhaps renaming this to `CIPHER_TRANSFORMATION` would be helpful.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ protected InputStream getInputStream(final File file) throws IOException {
+ final FileInputStream fis = new FileInputStream(file);
+ try {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ final int countIV = fis.read(iv);
+ if (countIV != SIZE_IV_AES_BYTES) {
+ throw new IOException(String.format(
+ "problem reading IV [expected=%d, actual=%d]", SIZE_IV_AES_BYTES, countIV));
+ }
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.DECRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherInputStream(fis, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ protected OutputStream getOutputStream(final File file) throws IOException {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ new SecureRandom().nextBytes(iv);
Review comment:
Should a new instance of `SecureRandom` be constructed on each method invocation, or can the instance be reused?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ protected InputStream getInputStream(final File file) throws IOException {
+ final FileInputStream fis = new FileInputStream(file);
+ try {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ final int countIV = fis.read(iv);
Review comment:
Perhaps renaming this to ivBytesRead would be helpful?
```suggestion
final int ivBytesRead = fis.read(iv);
```
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
Review comment:
If this constructor remains, it appears that this message was truncated.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
Review comment:
Is this constructor actually necessary? The instantiation code appears to locate the correct constructor that takes the `NiFiProperties` parameter, so could this be removed?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapManagerInitializationContext;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.StaticKeyProvider;
+import org.apache.nifi.util.NiFiProperties;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.logging.Logger;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test cases for {@link FileSystemSwapManagerEncrypt}.
+ */
+public class TestFileSystemSwapManagerEncrypt {
+ private static final Logger logger = Logger.getLogger(TestFileSystemSwapManagerEncrypt.class.getName());
+
+ /**
+ * Test a simple swap to disk / swap from disk operation. Configured to use {@link StaticKeyProvider}.
+ */
+ @Test
+ public void testConfig_StaticKeyProvider_SerdeSuccess() throws GeneralSecurityException, EncryptionException, IOException {
+ // use temp folder on filesystem to temporarily hold swap content (clean up after test)
+ final File folderRepository = Files.createTempDirectory(getClass().getSimpleName()).toFile();
+ logger.info(folderRepository.getPath());
+ folderRepository.deleteOnExit();
+ new File(folderRepository, "swap").deleteOnExit();
+
+ // configure a nifi properties for encrypted swap file
+ final Properties properties = new Properties();
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_PROVIDER_IMPLEMENTATION_CLASS, StaticKeyProvider.class.getName());
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_ID, NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY);
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY, StringUtils.repeat("00", 32));
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_DIRECTORY, folderRepository.getPath());
+ final NiFiProperties nifiProperties = NiFiProperties.createBasicNiFiProperties(null, properties);
+
+ // generate some flow file content to swap to disk
+ final List<FlowFileRecord> flowFiles = new ArrayList<>();
+ for (int i = 0; (i < 10_000); ++i) {
+ flowFiles.add(new StandardFlowFileRecord.Builder().id(i).build());
+ }
+
+ // setup for test case
+ final FlowFileSwapManager swapManager = createSwapManager(nifiProperties);
+ final String queueIdentifier = UUID.randomUUID().toString();
+ final FlowFileQueue flowFileQueue = Mockito.mock(FlowFileQueue.class);
+ when(flowFileQueue.getIdentifier()).thenReturn(queueIdentifier);
+
+ // swap out to disk; pull content back from disk
+ final String swapPath = swapManager.swapOut(flowFiles, flowFileQueue, "partition-1");
+ final SwapContents swapContents = swapManager.swapIn(swapPath, flowFileQueue);
+
+ // verify recovery of original content
+ final List<FlowFileRecord> flowFilesRecovered = swapContents.getFlowFiles();
+ Assert.assertEquals(flowFiles.size(), flowFilesRecovered.size());
+ Assert.assertTrue(flowFilesRecovered.containsAll(flowFiles));
+ Assert.assertTrue(flowFiles.containsAll(flowFilesRecovered));
+ }
+
+ /**
+ * Borrowed from "nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManager.java".
+ */
+ private FlowFileSwapManager createSwapManager(NiFiProperties nifiProperties)
+ throws IOException, GeneralSecurityException, EncryptionException {
+ final FlowFileRepository flowFileRepo = Mockito.mock(FlowFileRepository.class);
+ when(flowFileRepo.isValidSwapLocationSuffix(any())).thenReturn(true);
+ final FileSystemSwapManager swapManager = new FileSystemSwapManagerEncrypt(nifiProperties);
+ final ResourceClaimManager resourceClaimManager = new NopResourceClaimManager();
+
+ swapManager.initialize(new SwapManagerInitializationContext() {
+ @Override
+ public ResourceClaimManager getResourceClaimManager() {
+ return resourceClaimManager;
+ }
+
+ @Override
+ public FlowFileRepository getFlowFileRepository() {
+ return flowFileRepo;
+ }
+
+ @Override
+ public EventReporter getEventReporter() {
+ return EventReporter.NO_OP;
+ }
+ });
+
+ return swapManager;
+ }
+
+ /**
+ * Borrowed from "nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManager.java".
+ */
+ public static class NopResourceClaimManager implements ResourceClaimManager {
Review comment:
It looks like this class could be replaced with a mocked instance using Mockito.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapManagerInitializationContext;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.StaticKeyProvider;
+import org.apache.nifi.util.NiFiProperties;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.logging.Logger;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test cases for {@link FileSystemSwapManagerEncrypt}.
+ */
+public class TestFileSystemSwapManagerEncrypt {
+ private static final Logger logger = Logger.getLogger(TestFileSystemSwapManagerEncrypt.class.getName());
+
+ /**
+ * Test a simple swap to disk / swap from disk operation. Configured to use {@link StaticKeyProvider}.
+ */
+ @Test
+ public void testConfig_StaticKeyProvider_SerdeSuccess() throws GeneralSecurityException, EncryptionException, IOException {
+ // use temp folder on filesystem to temporarily hold swap content (clean up after test)
+ final File folderRepository = Files.createTempDirectory(getClass().getSimpleName()).toFile();
+ logger.info(folderRepository.getPath());
+ folderRepository.deleteOnExit();
+ new File(folderRepository, "swap").deleteOnExit();
+
+ // configure a nifi properties for encrypted swap file
+ final Properties properties = new Properties();
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_PROVIDER_IMPLEMENTATION_CLASS, StaticKeyProvider.class.getName());
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_ID, NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY);
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY, StringUtils.repeat("00", 32));
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_DIRECTORY, folderRepository.getPath());
+ final NiFiProperties nifiProperties = NiFiProperties.createBasicNiFiProperties(null, properties);
+
+ // generate some flow file content to swap to disk
+ final List<FlowFileRecord> flowFiles = new ArrayList<>();
+ for (int i = 0; (i < 10_000); ++i) {
+ flowFiles.add(new StandardFlowFileRecord.Builder().id(i).build());
+ }
+
+ // setup for test case
+ final FlowFileSwapManager swapManager = createSwapManager(nifiProperties);
+ final String queueIdentifier = UUID.randomUUID().toString();
+ final FlowFileQueue flowFileQueue = Mockito.mock(FlowFileQueue.class);
+ when(flowFileQueue.getIdentifier()).thenReturn(queueIdentifier);
+
+ // swap out to disk; pull content back from disk
+ final String swapPath = swapManager.swapOut(flowFiles, flowFileQueue, "partition-1");
+ final SwapContents swapContents = swapManager.swapIn(swapPath, flowFileQueue);
+
+ // verify recovery of original content
+ final List<FlowFileRecord> flowFilesRecovered = swapContents.getFlowFiles();
+ Assert.assertEquals(flowFiles.size(), flowFilesRecovered.size());
+ Assert.assertTrue(flowFilesRecovered.containsAll(flowFiles));
+ Assert.assertTrue(flowFiles.containsAll(flowFilesRecovered));
+ }
+
+ /**
+ * Borrowed from "nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManager.java".
+ */
+ private FlowFileSwapManager createSwapManager(NiFiProperties nifiProperties)
+ throws IOException, GeneralSecurityException, EncryptionException {
+ final FlowFileRepository flowFileRepo = Mockito.mock(FlowFileRepository.class);
+ when(flowFileRepo.isValidSwapLocationSuffix(any())).thenReturn(true);
+ final FileSystemSwapManager swapManager = new FileSystemSwapManagerEncrypt(nifiProperties);
+ final ResourceClaimManager resourceClaimManager = new NopResourceClaimManager();
+
+ swapManager.initialize(new SwapManagerInitializationContext() {
+ @Override
+ public ResourceClaimManager getResourceClaimManager() {
+ return resourceClaimManager;
+ }
+
+ @Override
+ public FlowFileRepository getFlowFileRepository() {
+ return flowFileRepo;
+ }
+
+ @Override
+ public EventReporter getEventReporter() {
+ return EventReporter.NO_OP;
+ }
+ });
Review comment:
Could this inline implementation be replaced with a Mockito instance of SwapManagerInitializationContext?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapManagerInitializationContext;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.StaticKeyProvider;
+import org.apache.nifi.util.NiFiProperties;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.logging.Logger;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test cases for {@link FileSystemSwapManagerEncrypt}.
+ */
+public class TestFileSystemSwapManagerEncrypt {
+ private static final Logger logger = Logger.getLogger(TestFileSystemSwapManagerEncrypt.class.getName());
+
+ /**
+ * Test a simple swap to disk / swap from disk operation. Configured to use {@link StaticKeyProvider}.
+ */
+ @Test
+ public void testConfig_StaticKeyProvider_SerdeSuccess() throws GeneralSecurityException, EncryptionException, IOException {
+ // use temp folder on filesystem to temporarily hold swap content (clean up after test)
+ final File folderRepository = Files.createTempDirectory(getClass().getSimpleName()).toFile();
+ logger.info(folderRepository.getPath());
+ folderRepository.deleteOnExit();
Review comment:
Is this sufficient to delete the contents of the directory?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.security.kms.CryptoUtils;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.KeyProvider;
+import org.apache.nifi.security.repository.RepositoryEncryptorUtils;
+import org.apache.nifi.security.repository.config.FlowFileRepositoryEncryptionConfiguration;
+import org.apache.nifi.util.NiFiProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.crypto.Cipher;
+import javax.crypto.CipherInputStream;
+import javax.crypto.CipherOutputStream;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.GCMParameterSpec;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Path;
+import java.security.GeneralSecurityException;
+import java.security.SecureRandom;
+
+/**
+ * <p>
+ * An implementation of the {@link FlowFileSwapManager} that swaps FlowFiles
+ * to/from local disk. The swap file is encrypted using AES/CTR, using the
+ * encryption key defined in nifi.properties.
+ * </p>
+ */
+@SuppressWarnings("unused") // instantiation via reflection in product
+public class FileSystemSwapManagerEncrypt extends FileSystemSwapManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(FileSystemSwapManagerEncrypt.class);
+
+ private final SecretKey secretKey;
+
+ /**
+ * Default no args constructor for service loading only.
+ */
+ public FileSystemSwapManagerEncrypt() {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ public FileSystemSwapManagerEncrypt(final NiFiProperties nifiProperties)
+ throws IOException, EncryptionException, GeneralSecurityException {
+ super(nifiProperties);
+ // acquire reference to FlowFileRepository key
+ final FlowFileRepositoryEncryptionConfiguration configuration = new FlowFileRepositoryEncryptionConfiguration(nifiProperties);
+ if (!CryptoUtils.isValidRepositoryEncryptionConfiguration(configuration)) {
+ logger.error("The flowfile repository encryption configuration is not valid (see above). Shutting down...");
+ throw new EncryptionException("The flowfile repository encryption configuration is not valid");
+ }
+ final KeyProvider keyProvider = RepositoryEncryptorUtils.validateAndBuildRepositoryKeyProvider(configuration);
+ this.secretKey = keyProvider.getKey(configuration.getEncryptionKeyId());
+ }
+
+ public FileSystemSwapManagerEncrypt(final Path flowFileRepoPath) {
+ throw new IllegalStateException("ctor not supported; nifi properties not available");
+ }
+
+ protected InputStream getInputStream(final File file) throws IOException {
+ final FileInputStream fis = new FileInputStream(file);
+ try {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ final int countIV = fis.read(iv);
+ if (countIV != SIZE_IV_AES_BYTES) {
+ throw new IOException(String.format(
+ "problem reading IV [expected=%d, actual=%d]", SIZE_IV_AES_BYTES, countIV));
+ }
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.DECRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherInputStream(fis, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ protected OutputStream getOutputStream(final File file) throws IOException {
+ final byte[] iv = new byte[SIZE_IV_AES_BYTES];
+ new SecureRandom().nextBytes(iv);
+ final FileOutputStream fos = new FileOutputStream(file);
+ fos.write(iv);
+ try {
+ final Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+ cipher.init(Cipher.ENCRYPT_MODE, secretKey, new GCMParameterSpec(SIZE_TAG_GCM_BITS, iv));
+ return new CipherOutputStream(fos, cipher);
+ } catch (GeneralSecurityException e) {
+ throw new IOException(e);
+ }
+ }
+
+ private static final String ALGORITHM = "AES";
+ private static final String TRANSFORMATION = "AES/GCM/NoPadding";
+ private static final int SIZE_IV_AES_BYTES = 16;
+ private static final int SIZE_TAG_GCM_BITS = 128;
Review comment:
Recommend moving these values to the top of the class following general existing conventions.
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapManagerInitializationContext;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.StaticKeyProvider;
+import org.apache.nifi.util.NiFiProperties;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.logging.Logger;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test cases for {@link FileSystemSwapManagerEncrypt}.
+ */
+public class TestFileSystemSwapManagerEncrypt {
+ private static final Logger logger = Logger.getLogger(TestFileSystemSwapManagerEncrypt.class.getName());
+
+ /**
+ * Test a simple swap to disk / swap from disk operation. Configured to use {@link StaticKeyProvider}.
+ */
+ @Test
+ public void testConfig_StaticKeyProvider_SerdeSuccess() throws GeneralSecurityException, EncryptionException, IOException {
+ // use temp folder on filesystem to temporarily hold swap content (clean up after test)
+ final File folderRepository = Files.createTempDirectory(getClass().getSimpleName()).toFile();
+ logger.info(folderRepository.getPath());
+ folderRepository.deleteOnExit();
+ new File(folderRepository, "swap").deleteOnExit();
+
+ // configure a nifi properties for encrypted swap file
+ final Properties properties = new Properties();
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_PROVIDER_IMPLEMENTATION_CLASS, StaticKeyProvider.class.getName());
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_ID, NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY);
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY, StringUtils.repeat("00", 32));
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_DIRECTORY, folderRepository.getPath());
+ final NiFiProperties nifiProperties = NiFiProperties.createBasicNiFiProperties(null, properties);
+
+ // generate some flow file content to swap to disk
+ final List<FlowFileRecord> flowFiles = new ArrayList<>();
+ for (int i = 0; (i < 10_000); ++i) {
+ flowFiles.add(new StandardFlowFileRecord.Builder().id(i).build());
+ }
+
+ // setup for test case
Review comment:
This comment and several others do not appear to add much in the way of clarification to the test behavior, should they be removed?
##########
File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/TestFileSystemSwapManagerEncrypt.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.FlowFileSwapManager;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapManagerInitializationContext;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.security.kms.EncryptionException;
+import org.apache.nifi.security.kms.StaticKeyProvider;
+import org.apache.nifi.util.NiFiProperties;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.logging.Logger;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test cases for {@link FileSystemSwapManagerEncrypt}.
+ */
+public class TestFileSystemSwapManagerEncrypt {
+ private static final Logger logger = Logger.getLogger(TestFileSystemSwapManagerEncrypt.class.getName());
+
+ /**
+ * Test a simple swap to disk / swap from disk operation. Configured to use {@link StaticKeyProvider}.
+ */
+ @Test
+ public void testConfig_StaticKeyProvider_SerdeSuccess() throws GeneralSecurityException, EncryptionException, IOException {
+ // use temp folder on filesystem to temporarily hold swap content (clean up after test)
+ final File folderRepository = Files.createTempDirectory(getClass().getSimpleName()).toFile();
+ logger.info(folderRepository.getPath());
+ folderRepository.deleteOnExit();
+ new File(folderRepository, "swap").deleteOnExit();
+
+ // configure a nifi properties for encrypted swap file
+ final Properties properties = new Properties();
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_PROVIDER_IMPLEMENTATION_CLASS, StaticKeyProvider.class.getName());
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY_ID, NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY);
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_ENCRYPTION_KEY, StringUtils.repeat("00", 32));
+ properties.put(NiFiProperties.FLOWFILE_REPOSITORY_DIRECTORY, folderRepository.getPath());
+ final NiFiProperties nifiProperties = NiFiProperties.createBasicNiFiProperties(null, properties);
+
+ // generate some flow file content to swap to disk
+ final List<FlowFileRecord> flowFiles = new ArrayList<>();
+ for (int i = 0; (i < 10_000); ++i) {
Review comment:
Is it necessary to create this many files? Recommend a much smaller number for test performance unless there is a reason for the larger number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org