You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by jo...@apache.org on 2022/05/12 19:51:33 UTC

[nifi] branch support/nifi-1.16 updated (ad545ac45a -> 7b9201c856)

This is an automated email from the ASF dual-hosted git repository.

joewitt pushed a change to branch support/nifi-1.16
in repository https://gitbox.apache.org/repos/asf/nifi.git


    from ad545ac45a NIFI-10009 - bump Kudu client from 1.15 to 1.16
     new e390f87e2b NIFI-10003 - NiFi build fails in UTC+ time zones
     new ad75c80f32 NIFI-10023: Ensure that any files that are archived upon startup properly increment the archival count in the content repo
     new 04a40d4e49 NIFI-10020 This closes #6039. Corrected ClassNotFoundException in EmailNotificationService
     new 7b9201c856 NIFI-10024 This closes #6041. Upgraded Netty from 4.1.74 to 4.1.77

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 nifi-bootstrap/pom.xml                             |  6 +++
 .../repository/FileSystemRepository.java           | 29 +++++++++++--
 .../repository/TestFileSystemRepository.java       | 50 ++++++++++++++++++++++
 .../src/test/resources/conf/nifi.properties        |  2 +
 .../nifi-scripting-processors/pom.xml              |  2 +-
 pom.xml                                            |  2 +-
 6 files changed, 86 insertions(+), 5 deletions(-)


[nifi] 04/04: NIFI-10024 This closes #6041. Upgraded Netty from 4.1.74 to 4.1.77

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joewitt pushed a commit to branch support/nifi-1.16
in repository https://gitbox.apache.org/repos/asf/nifi.git

commit 7b9201c85644f3e75db672078f723995dcca7646
Author: exceptionfactory <ex...@apache.org>
AuthorDate: Thu May 12 14:23:53 2022 -0500

    NIFI-10024 This closes #6041. Upgraded Netty from 4.1.74 to 4.1.77
    
    Signed-off-by: Joe Witt <jo...@apache.org>
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index ea4278658c..6eaeea037b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -128,7 +128,7 @@
         <logback.version>1.2.11</logback.version>
         <mockito.version>3.11.2</mockito.version>
         <netty.3.version>3.10.6.Final</netty.3.version>
-        <netty.4.version>4.1.74.Final</netty.4.version>
+        <netty.4.version>4.1.77.Final</netty.4.version>
         <spring.version>5.3.19</spring.version>
         <spring.security.version>5.6.3</spring.security.version>
         <h2.version>2.1.210</h2.version>


[nifi] 03/04: NIFI-10020 This closes #6039. Corrected ClassNotFoundException in EmailNotificationService

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joewitt pushed a commit to branch support/nifi-1.16
in repository https://gitbox.apache.org/repos/asf/nifi.git

commit 04a40d4e49ea65f16d6c0f9b2ee43ccf40d9ed1e
Author: exceptionfactory <ex...@apache.org>
AuthorDate: Thu May 12 13:12:08 2022 -0500

    NIFI-10020 This closes #6039. Corrected ClassNotFoundException in EmailNotificationService
    
    - Added activation-api runtime dependency to nifi-bootstrap
    
    Signed-off-by: Joe Witt <jo...@apache.org>
---
 nifi-bootstrap/pom.xml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/nifi-bootstrap/pom.xml b/nifi-bootstrap/pom.xml
index e408349090..82984c6d29 100644
--- a/nifi-bootstrap/pom.xml
+++ b/nifi-bootstrap/pom.xml
@@ -72,6 +72,12 @@ language governing permissions and limitations under the License. -->
             <artifactId>jakarta.mail</artifactId>
             <version>2.0.1</version>
         </dependency>
+        <dependency>
+            <groupId>jakarta.activation</groupId>
+            <artifactId>jakarta.activation-api</artifactId>
+            <version>2.0.1</version>
+            <scope>runtime</scope>
+        </dependency>
         <dependency>
             <groupId>org.apache.nifi</groupId>
             <artifactId>nifi-expression-language</artifactId>


[nifi] 02/04: NIFI-10023: Ensure that any files that are archived upon startup properly increment the archival count in the content repo

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joewitt pushed a commit to branch support/nifi-1.16
in repository https://gitbox.apache.org/repos/asf/nifi.git

commit ad75c80f32c17d061650c64722a640086bfdc1ec
Author: Mark Payne <ma...@hotmail.com>
AuthorDate: Thu May 12 11:10:36 2022 -0400

    NIFI-10023: Ensure that any files that are archived upon startup properly increment the archival count in the content repo
    
    Signed-off-by: Matthew Burgess <ma...@apache.org>
    
    This closes #6037
---
 .../repository/FileSystemRepository.java           | 29 +++++++++++--
 .../repository/TestFileSystemRepository.java       | 50 ++++++++++++++++++++++
 .../src/test/resources/conf/nifi.properties        |  2 +
 3 files changed, 78 insertions(+), 3 deletions(-)

diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java
index 4bebb44bbe..755bef032a 100644
--- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java
+++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java
@@ -504,11 +504,11 @@ public class FileSystemRepository implements ContentRepository {
 
         final ResourceClaim resourceClaim = resourceClaimManager.newResourceClaim(containerName, sectionName, id, false, false);
         if (resourceClaimManager.getClaimantCount(resourceClaim) == 0) {
-            removeIncompleteContent(fileToRemove);
+            removeIncompleteContent(fileToRemove, containerName);
         }
     }
 
-    private void removeIncompleteContent(final Path fileToRemove) {
+    private void removeIncompleteContent(final Path fileToRemove, final String containerName) {
         String fileDescription = null;
         try {
             fileDescription = fileToRemove.toFile().getAbsolutePath() + " (" + Files.size(fileToRemove) + " bytes)";
@@ -520,7 +520,16 @@ public class FileSystemRepository implements ContentRepository {
 
         try {
             if (archiveData) {
-                archive(fileToRemove);
+                final boolean archived = archive(fileToRemove);
+
+                if (archived) {
+                    final ContainerState containerState = containerStateMap.get(containerName);
+                    if (containerState == null) {
+                        LOG.warn("Failed to increment container's archive count for {} because container {} could not be found", fileToRemove.toFile(), containerName);
+                    } else {
+                        containerState.incrementArchiveCount();
+                    }
+                }
             } else {
                 Files.delete(fileToRemove);
             }
@@ -531,6 +540,16 @@ public class FileSystemRepository implements ContentRepository {
         }
     }
 
+    // Visible for testing
+    long getArchiveCount(String containerName) {
+        final ContainerState containerState = containerStateMap.get(containerName);
+        if (containerState == null) {
+            throw new IllegalArgumentException("No container exists with name " + containerName);
+        }
+
+        return containerState.getArchiveCount();
+    }
+
     @Override
     public boolean isActiveResourceClaimsSupported() {
         return true;
@@ -1769,6 +1788,10 @@ public class FileSystemRepository implements ContentRepository {
             archivedFileCount.incrementAndGet();
         }
 
+        public long getArchiveCount() {
+            return archivedFileCount.get();
+        }
+
         public void decrementArchiveCount() {
             archivedFileCount.decrementAndGet();
         }
diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestFileSystemRepository.java b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestFileSystemRepository.java
index e05938ac93..db762bfa08 100644
--- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestFileSystemRepository.java
+++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestFileSystemRepository.java
@@ -55,6 +55,7 @@ import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
 import java.util.Locale;
+import java.util.Map;
 import java.util.Random;
 import java.util.concurrent.TimeUnit;
 
@@ -140,6 +141,55 @@ public class TestFileSystemRepository {
         assertTrue(repository.isArchived(Paths.get("a/b/c/archive/1.txt")));
     }
 
+    @Test
+    public void testUnreferencedFilesAreArchivedOnCleanup() throws IOException {
+        final Map<String, Path> containerPaths = nifiProperties.getContentRepositoryPaths();
+        assertTrue(containerPaths.size() > 0);
+
+        for (final Map.Entry<String, Path> entry : containerPaths.entrySet()) {
+            final String containerName = entry.getKey();
+            final Path containerPath = entry.getValue();
+
+            final Path section1 = containerPath.resolve("1");
+            final Path file1 = section1.resolve("file-1");
+            Files.write(file1, "hello".getBytes(), StandardOpenOption.CREATE);
+
+            // Should be nothing in the archive at this point
+            assertEquals(0, repository.getArchiveCount(containerName));
+
+            // When we cleanup, we should see one file moved to archive
+            repository.cleanup();
+            assertEquals(1, repository.getArchiveCount(containerName));
+        }
+    }
+
+    @Test
+    public void testAlreadyArchivedFilesCounted() throws IOException {
+        // We want to make sure that the initialization code counts files in archive, so we need to create a new FileSystemRepository to do this.
+        repository.shutdown();
+
+        final Map<String, Path> containerPaths = nifiProperties.getContentRepositoryPaths();
+        assertTrue(containerPaths.size() > 0);
+
+        for (final Path containerPath : containerPaths.values()) {
+            final Path section1 = containerPath.resolve("1");
+            final Path archive = section1.resolve("archive");
+            Files.createDirectories(archive);
+
+            for (int i=0; i < 3; i++) {
+                final Path file1 = archive.resolve("file-" + i);
+                Files.write(file1, "hello".getBytes(), StandardOpenOption.CREATE);
+            }
+        }
+
+        repository = new FileSystemRepository(nifiProperties);
+
+        for (final String containerName : containerPaths.keySet()) {
+            assertEquals(3, repository.getArchiveCount(containerName));
+        }
+    }
+
+
     @Test
     public void testContentNotFoundExceptionThrownIfResourceClaimTooShort() throws IOException {
         final File contentFile = new File("target/content_repository/0/0.bin");
diff --git a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/conf/nifi.properties b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/conf/nifi.properties
index 3490ee59eb..1427ae8607 100644
--- a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/conf/nifi.properties
+++ b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/conf/nifi.properties
@@ -48,6 +48,8 @@ nifi.swap.out.threads=4
 nifi.content.claim.max.appendable.size=1 MB
 nifi.content.claim.max.flow.files=100
 nifi.content.repository.directory.default=./target/content_repository
+nifi.content.repository.archive.enabled=true
+nifi.content.repository.archive.max.usage.percentage=90%
 
 # Provenance Repository Properties
 nifi.provenance.repository.storage.directory=./target/provenance_repository


[nifi] 01/04: NIFI-10003 - NiFi build fails in UTC+ time zones

Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

joewitt pushed a commit to branch support/nifi-1.16
in repository https://gitbox.apache.org/repos/asf/nifi.git

commit e390f87e2bc93561b39ec4971c179d2c4cd20c20
Author: Paul Grey <gr...@yahoo.com>
AuthorDate: Thu May 12 11:27:05 2022 -0400

    NIFI-10003 - NiFi build fails in UTC+ time zones
    
    Signed-off-by: Matthew Burgess <ma...@apache.org>
    
    This closes #6038
---
 .../nifi-scripting-bundle/nifi-scripting-processors/pom.xml             | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/pom.xml b/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/pom.xml
index 8ca1385a5a..e68ec5ee9f 100644
--- a/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/pom.xml
+++ b/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/pom.xml
@@ -77,7 +77,7 @@
         </dependency>
         <dependency>
             <groupId>org.python</groupId>
-            <artifactId>jython-slim</artifactId>
+            <artifactId>jython-standalone</artifactId>
             <version>2.7.2</version>
         </dependency>
         <dependency>