You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "simonbence (via GitHub)" <gi...@apache.org> on 2023/07/17 14:11:44 UTC

[GitHub] [nifi] simonbence commented on a diff in pull request #7484: NIFI-11817: Improve ListHDFS extensibility

simonbence commented on code in PR #7484:
URL: https://github.com/apache/nifi/pull/7484#discussion_r1265426467


##########
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java:
##########
@@ -263,24 +267,35 @@ public void onTrigger(final ProcessContext context, final ProcessSession session
 
             final FileStatusManager fileStatusManager = new FileStatusManager(latestTimestamp, latestFiles);
             final Path rootPath = getNormalizedPath(context, DIRECTORY);
-            final FileStatusIterable fileStatuses = new FileStatusIterable(rootPath, recursive, hdfs, getUserGroupInformation());
+            final FileStatusIterable fileStatusIterable = new FileStatusIterable(rootPath, recursive, hdfs, getUserGroupInformation());
 
             final Long minAgeProp = context.getProperty(MINIMUM_FILE_AGE).asTimePeriod(TimeUnit.MILLISECONDS);
             final long minimumAge = (minAgeProp == null) ? Long.MIN_VALUE : minAgeProp;
             final Long maxAgeProp = context.getProperty(MAXIMUM_FILE_AGE).asTimePeriod(TimeUnit.MILLISECONDS);
             final long maximumAge = (maxAgeProp == null) ? Long.MAX_VALUE : maxAgeProp;
 
-            final HadoopFileStatusWriter writer;
-            if (writerFactory == null) {
-                writer = new FlowFileObjectWriter(session, fileStatuses, minimumAge, maximumAge, pathFilter, fileStatusManager, latestTimestamp, latestFiles);
-            } else {
-                writer = new RecordObjectWriter(session, fileStatuses, minimumAge, maximumAge, pathFilter, fileStatusManager, latestTimestamp,
-                        latestFiles, writerFactory, getLogger());
-            }
+            final HadoopWriterContext hadoopWriterContext = HadoopWriterContext.builder()
+                    .session(session)
+                    .successRelationship(getSuccessRelationship())
+                    .fileStatusIterable(fileStatusIterable)
+                    .fileStatusManager(fileStatusManager)
+                    .pathFilter(pathFilter)
+                    .minimumAge(minimumAge)
+                    .maximumAge(maximumAge)
+                    .previousLatestTimestamp(latestTimestamp)
+                    .previousLatestFiles(latestFiles)
+                    .writerFactory(writerFactory)
+                    .hdfsPrefix(getAttributePrefix())
+                    .logger(getLogger())
+                    .build();
+
+            final HadoopFileStatusWriter writer = (writerFactory == null)

Review Comment:
   Adding a builder looks to be a great step forward! It could be even more beneficial for us if the builder would not only build the context but the writer and the actual writer instance creation would be hidden behind the `build` method. I think, adding an extra method which can help the builder decide what kind of output we expect (maybe a small enum) would help with this and this way also the context and the writer implementations might remain unexposed.



##########
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/util/writer/HadoopWriterContext.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hadoop.util.writer;
+
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processors.hadoop.util.FileStatusIterable;
+import org.apache.nifi.processors.hadoop.util.FileStatusManager;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+
+import java.util.List;
+
+public class HadoopWriterContext {
+
+    private final ProcessSession session;
+    private final Relationship successRelationship;
+    private final FileStatusIterable fileStatusIterable;
+    private final FileStatusManager fileStatusManager;
+    private final PathFilter pathFilter;
+    private final long minimumAge;
+    private final long maximumAge;
+    private final long previousLatestTimestamp;
+    private final List<String> previousLatestFiles;
+    private final RecordSetWriterFactory writerFactory;
+    private final String hdfsPrefix;
+    private final ComponentLog logger;
+
+
+    public HadoopWriterContext(final ProcessSession session,
+                               final Relationship successRelationship,
+                               final FileStatusIterable fileStatusIterable,
+                               final FileStatusManager fileStatusManager,
+                               final PathFilter pathFilter,
+                               final long minimumAge,
+                               final long maximumAge,
+                               final long previousLatestTimestamp,
+                               final List<String> previousLatestFiles,
+                               final RecordSetWriterFactory writerFactory,
+                               final String hdfsPrefix,
+                               final ComponentLog logger) {
+        this.session = session;
+        this.successRelationship = successRelationship;
+        this.fileStatusIterable = fileStatusIterable;
+        this.fileStatusManager = fileStatusManager;
+        this.pathFilter = pathFilter;
+        this.minimumAge = minimumAge;
+        this.maximumAge = maximumAge;
+        this.previousLatestTimestamp = previousLatestTimestamp;
+        this.previousLatestFiles = previousLatestFiles;
+        this.writerFactory = writerFactory;
+        this.hdfsPrefix = hdfsPrefix;
+        this.logger = logger;
+    }
+
+    public static Builder builder() {
+        return new Builder();
+    }
+
+    public static class Builder {
+        private ProcessSession session;
+        private FileStatusIterable fileStatusIterable;
+        private long minimumAge;
+        private long maximumAge;
+        private PathFilter pathFilter;
+        private FileStatusManager fileStatusManager;
+        private long previousLatestTimestamp;
+        private List<String> previousLatestFiles;
+        private Relationship successRelationship;
+        private RecordSetWriterFactory writerFactory;
+        private ComponentLog logger;
+        private String hdfsPrefix;
+
+        public Builder session(final ProcessSession session) {
+            this.session = session;
+            return this;
+        }
+
+        public Builder fileStatusIterable(final FileStatusIterable fileStatusIterable) {
+            this.fileStatusIterable = fileStatusIterable;
+            return this;
+        }
+
+        public Builder minimumAge(final long minimumAge) {
+            this.minimumAge = minimumAge;
+            return this;
+        }
+
+        public Builder maximumAge(final long maximumAge) {
+            this.maximumAge = maximumAge;
+            return this;
+        }
+
+        public Builder pathFilter(final PathFilter pathFilter) {
+            this.pathFilter = pathFilter;
+            return this;
+        }
+
+        public Builder fileStatusManager(final FileStatusManager fileStatusManager) {
+            this.fileStatusManager = fileStatusManager;
+            return this;
+        }
+
+        public Builder previousLatestTimestamp(final long previousLatestTimestamp) {
+            this.previousLatestTimestamp = previousLatestTimestamp;
+            return this;
+        }
+
+        public Builder previousLatestFiles(final List<String> previousLatestFiles) {
+            this.previousLatestFiles = previousLatestFiles;
+            return this;
+        }
+
+        public Builder successRelationship(final Relationship successRelationship) {
+            this.successRelationship = successRelationship;
+            return this;
+        }
+
+        public Builder writerFactory(final RecordSetWriterFactory writerFactory) {
+            this.writerFactory = writerFactory;
+            return this;
+        }
+
+        public Builder logger(final ComponentLog logger) {
+            this.logger = logger;
+            return this;
+        }
+
+        public Builder hdfsPrefix(final String hdfsPrefix) {
+            this.hdfsPrefix = hdfsPrefix;
+            return this;
+        }
+
+        public HadoopWriterContext build() {
+            return new HadoopWriterContext(session, successRelationship, fileStatusIterable, fileStatusManager, pathFilter, minimumAge, maximumAge,

Review Comment:
   I would suggest to add some validation here which could check if all the mandatory attributes are set. Like for example: checking if `session` is not null.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org